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A SYSTEM METHOD AND ARTICLE OF MANUFACTURE FOR 
CREATING INTERACTIVE SIMULATIONS UTILIZING A REMOTE 

KNOWLEDGE BASE 

5 FIELD OF THE INVENTION 

The present invention relates to education systems and more particularly to a rule 
based tutorial system that utilizes business simulations of actual environments to 
teach new skills. 

10 BACKGROUND OF THE INVENTION 

When building a knowledge based system or expert system, at least two disciphnes 
are necessary to properly construct the rules that drive the knowledge base, the 
discipline of the knowledge engineer and the knowledge of the expert. The domain 
expert has knowledge of the domain or field of use of the expert system. For 

15 example, the domain expert of an expert for instructing students in an automotive 
manufacturing facility might be a process control engineer while the domain expert 
for a medical instruction system might be a doctor or a nurse. The knowledge 
engineer is a person that understands the expert system and utilizes the expert's 
knowledge to create an apphcation for the system. In many instances, the 

20 knowledge engineer and domain expert are separate people who have to collaborate 
to construct the expert system. 

Typically, this collaboration takes the form of the knowledge engineer asking 
questions of the domain expert and incorporating the answers to these questions into 

25 the design of the system. This approach is labor intensive, slow and error prone. 
The coordination of the two separate disciplines may lead to problems. Although 
the knowledge engineer can transcribe input from the expert utiUzing videotape, 
audio tape, text and other sources, efforts from people of both disciplines have to be 
expended. Further, if the knowledge engineer does not ask the right questions or 

30 asks the questions in an incorrect way, the information utilized to design the 

knowledge base could be incorrect. Feedback to the knowledge engineer from the 
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expert system is often not available in prior art system until the construction is 
completed. With conventional system, there is a time consuming feedback loop that 
ties together various processes from knowledge acquisition to validation. 

5 Educational systems utilizing an expert system component often suffer from a lack 
of motivational aspects that result in a user becoming bored or ceasing to complete a 
training program. Current training programs utilize static, hard-coded feedback with 
some linear video and graphics used to add visual appeal and illustrate concepts. 
These systems typically support one "correct" answer and navigation through the 
10 system is only supported through a single defined path which results in a two- 
dimensional generic interaction, with no business model support and a single 
feedback to the learner of correct or incorrect based on the selected response. 
Current tutorial systems do not architect real business simulations into the rules to 
provide a creative learning environment to a user. 

15 

SUMMARY OF THE INVENTION 

According to a broad aspect of a preferred embodiment of the invention, a goal 
based learning system utilizes a rule based expert training system to provide a 
cognitive educational experience. 

20 

A system is disclosed that provides a goal based learning system utilizing a rule 
based expert training system to provide a cognitive educational experience. The 
system provides the user with a simulated environment that presents a training 
opportunity to understand and solve optimally. Mistakes are noted and remedial 

25 educational material presented dynamically to build the necessary skills that a user 
requires for success in the business endeavor. The difficulty level is automatically 
adjusted to the student's skill level. The system utilizes an artificial intelligence 
engine driving individualized and dynamic feedback with synchronized audio, 
video, graphics and animation used to simulate real-world environment and 

30 interactions. Multiple "correct" answers are integrated into the learning system to 
allow individualized learning experiences in which navigation through the system is 
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at a pace controlled by the learner. A robust business model provides support for 
realistic activities and allows a user to experience real world consequences for their 
actions and decisions and entails realtime decision-making and synthesis of the 
educational material. A dynamic feedback system is utilized that narrowly tailors 
5 feedback and focuses it based on the performance and characteristics of the student 
to assist the student in reaching a predefined goal 



DESCRIPTION OF THE DRAWINGS 

The foregoing and other objects, aspects and advantages are better understood fi:om 
1 0 the following detailed description of a preferred embodiment of the invention with 
reference to the drawings, in which: 

^0 Figure 1 is a block diagram of a representative hardware environment in accordance 

IJ1 with a preferred embodiment; 

S 15 

;p Figure 2 is a block diagram of a system architecture in accordance with a preferred 

m embodiment; 

11 Figure 3 depicts the timeUne and relative resource requirements for each phase of 

O 

20 development for a typical application development in accordance with a preferred 
embodiment; 



Figure 4 depicts the potential savings in both functional and technical tasks in 
accordance with a preferred embodiment; 

25 

Figure 5 illustrates commonalties in accordance with a preferred embodiment; 

Figure 6 illustrates a development architecture approach in accordance with a 
preferred embodiment; 
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Figure 7 illustrates a small segment of a domain model for claims handlers in the 
auto insurance industry in accordance with a preferred embodiment; 

Figure 8 illustrates an instantiated domain model in accordance with a preferred 
5 embodiment; 

Figure 9 illustrates an insurance underwriting profile in accordance with a preferred 
embodiment; 

10 Figure 10 illustrates a transformation component in accordance with a preferred 
embodiment; 

Figure 11 illustrates the use of a toolbar to navigate and access application level 
features in accordance with a preferred embodiment; 

Figure 12 is a GBS display in accordance with a preferred embodiment; 

Figure 13 is a feedback display in accordance with a preferred embodiment; 

Figure 14 illustrates a display in which a student has made some mistakes in 
accordance with a preferred embodiment; 

Figure 15 illustrates a journal entry simulation in accordance with a preferred 
embodiment; 
25 

Figure 16 illustrates a simulated Bell Phone Bill journal entry in accordance with a 
preferred embodiment; 

Figure 17 illustrates a feedback display in accordance with a preferred embodiment; 

30 
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Figures 18 and 19 illustrate a feedback display in accordance with a preferred 
embodiment; 

Figure 20 illustrates a feedback display in accordance with a preferred embodiment; 

5 

Figure 21 illustrates a simulation display in accordance with a preferred 
embodiment; 

Figure 22 illustrates the steps of the first scenario in accordance with a preferred 
10 embodiment; 

% Figure 23 and 24 illustrate the steps associated with a build scenario in accordance 

0 with a preferred embodnnent; 

11 

^ 15 Figure 25 illustrates how the tool suite supports student administration in accordance 
P with a preferred embodiment; 

Figure 26 illustrates a suite to support a student interaction in accordance with a 
II preferred embodiment; 

t 20 

Figure 27 illustrates the remediation process in accordance with a preferred 
embodiment; 

Figure 28 illustrates a display of journalization transactions in accordance with a 
25 preferred embodiment; 

Figure 29 illustrates the objects for the journalization task in accordance with a 
preferred embodiment; 
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Figiure 30 illustrates the mapping of a source item to a target item in accordance with 
a preferred embodiment; 
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Figure 31 illustrates target group bundles in accordance with a preferred 
embodiment; 

5 Figure 32 illustrates a TargetGroup Hierarchy in accordance with a preferred 
embodiment; 

Figure 33 illustrates a small section the amount of feedback in accordance with a 
preferred embodiment; 

10 

Figure 34 illustrates an analysis of rules in accordance with a preferred embodiment; 

Figure 35 illustrates a feedback selection in accordance with a preferred 
embodiment; 

15 

Figure 36 is a flowchart of the feedback logic in accordance with a preferred 
embodiment; 

Figure 37 illustrates an example of separating out some mistakes for the interface to 
20 catch and others for the ICAT to catch has positive and negative impacts in 
accordance with a preferred embodiment; 

Figure 38 is a block diagram of the hierarchical relationship of a transaction in 
accordance with a preferred embodiment; 

25 

Figure 39 is a block diagram illustrating the feedback hierarchy in accordance with a 
preferred embodiment; 

Figure 40 is a block diagram illustrating how the simulation engine is architected 
30 into a preferred embodiment of the invention; 
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Figure 41 is a block diagram setting forth the architecture of a simulation model in 
accordance with a preferred embodiment; 

5 Figure 42 illustrates the arithmetic steps in accordance with a preferred embodiment; 

Figure 43 illustrates a drag & drop input operation in accordance with a preferred 
embodiment; 

10 Figure 44 illustrates hst object processing in accordance with a preferred 
embodiment; 

Figure 45 illustrates the steps for configuring a simulation in accordance with a 
preferred embodiment; 

15 

Figure 46 illustrates a distinct output in accordance with a preferred embodiment; 

Figure 47 is a block diagram presenting the detailed architecture of a system 
dynamics model in accordance with a preferred embodiment; 

20 

Figure 48 is graphical representation of the object model which is utiHzed to 
instantiate the system dynamic engine in accordance with a preferred embodiment. 

Figure 49 is a PInput Cell for a simulation model in accordance with a preferred 
25 embodiment; 

Figure 50 is a PInput backup cell in a simulation model in accordance with a 
preferred embodiment; 

30 Figure 51 is a display illustrating a POutput cell in accordance with a preferred 
embodiment. The steps required to configure the POutput are presented below; 



-8- 

Figure 52 is an overview diagram of the logic utilized for initial configuration in 
accordance with a preferred embodiment; 

Figure 53 is a display of the source item and target configuration in accordance with 
5 a preferred embodiment; 

Figure 54 is a display of video information in accordance with a preferred 
embodiment; 

10 Figure 55 illustrates a display depicting configured rules in accordance with a 
preferred embodiment; 

Figure 56 illustrates feedback for configured rules in accordance with a preferred 
embodiment; 

15 

Figure 57 illustrates a display with follow-up configuration questions in accordance 
with a preferred embodiment; 

Figure 58 illustrates configuration of aggregate rules in accordance with a preferred 
20 embodiment; 

Figure 59 illustrates a set of coach items in accordance with a preferred embodiment; 

Figure 60 is an ICA Meeting Configuration tool display in accordance with a 
25 preferred embodiment; 

Figure 61 illustrates an ICA utiUty in accordance with a preferred embodiment; 

Figure 62 illustrates a configuration utility display in accordance with a preferred 
30 embodiment; 
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Figure 63 illustrates an object editor toolbar in accordance with a preferred 
embodiment; 

Figure 64 illustrates the seven areas that can be configured for a simulation in 
5 accordance with a preferred embodiment; 

Figure 65 illustrates a display that defines inputs in accordance with a preferred 
embodiment; 

10 Figure 66 illustrates a list editor in accordance with a preferred embodiment; 

Figure 67A illustrates a define student display in accordance with a preferred 
embodiment; 

15 Figure 67B illustrates a ControlSourceltem display in accordance with a preferred 
embodiment; 

Figure 68 illustrates a simulation workbench in accordance with a preferred 
embodiment; 

20 

Figure 69 illustrates an object viewer in accordance with a preferred embodiment. 
As shown in 

Figure 70 illustrates an Object Viewer Configuration in an Utilities menu in 
25 accordance with a preferred embodiment; 

Figure 71 illustrates a log viewer in accordance with a preferred embodiment; 

Figure 72 illustrates a Doc Maker display in accordance with a preferred 
30 embodiment; 
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Figure 73 illustrates a Feedback display in accordance with a preferred embodiment; 

Figure 74 is an object editor display that illustrates the use of references in 
accordance with a preferred embodiment; 

5 

Figure 75 presents the detailed design of smart spreadsheets in accordance with a 
preferred embodiment; 

Figure 76 presents the assembly of a telephone operator training simulation in 
1 0 accordance with a preferred embodiment; 

Figure 77 presents the domain expert's work station utihzed to assemble a 
simulation in accordance with a preferred embodiment; 

15 Figure 78 presents multiple domain expert's work stations linked/networked to 
collaborate on the assembly of a simulation in accordance with a preferred 
embodiment; 

Figure 79 presents the detailed flowchart of a telephone operator training simulation 
20 in accordance with a preferred embodiment; 

Figure 80 presents a user training station linked/networked to the simulation server 
in accordance with a preferred embodiment; 

25 Figure 81 presents a detailed flowchart of a user query of the knowledge base in 
accordance with a preferred embodiment; 

Figure 82 presents an example of feedback from a coach in accordance with a 
preferred embodiment; 

30 
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Figure 83 presents multiple user's training stations linked/networked to collaborate 
in the execution of a simulation in accordance with a preferred embodiment; 

Figure 84 is a block diagram of a system environment in accordance with a preferred 
5 embodiment; 

Figure 85 is a block diagram of a virtual consulting channel in accordance with a 
preferred embodiment; 

10 Figures 86 and 87 are data structures for a virtual consulting environment in 
accordance with a preferred embodiment; and 

Figures 88 - 99 are flowcharts of a virtual university system in accordance with a 
preferred embodiment, 

15 

DETAILED DESCRIPTION 

A preferred embodiment of a system in accordance with the present invention is 
preferably practiced in the context of a personal computer such as an IBM 

20 compatible personal computer, Apple Macintosh computer or UNIX based 

workstation, A representative hardware environment is depicted in Figure 1, which 
illustrates a typical hardware configuration of a workstation in accordance with a 
preferred embodiment having a central processing unit 110, such as a 
microprocessor, and a number of other units interconnected via a system bus 112. 

25 The workstation shown in Figure 1 includes a Random Access Memory (RAM) 114, 
Read Only Memory (ROM) 116, an I/O adapter 118 for coimecting peripheral 
devices such as disk storage units 120 to the bus 112, a user interface adapter 122 for 
connecting a keyboard 124, a mouse 126, a speaker 128, a microphone 132, and/or 
other user interface devices such as a touch screen (not shown) to the bus 112, 

30 communication adapter 134 for connecting the workstation to a communication 
network (e.g., a data processing network) and a display adapter 136 for connecting 
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the bus 112 to a display device 138, The workstation typically has resident thereon 
an operating system such as the Microsoft Windows NT or Windows/95 Operating 
System (OS), the IBM OS/2 operating system, the MAC OS, or UNIX operating 
system. Those skilled in the art will appreciate that the present invention may also 
5 be implemented on platforms and operating systems other than those mentioned. 

A preferred embodiment is written using JAVA, C, and the C++ language and 
utilizes object oriented programming methodology. Object oriented programming 
(OOP) has become increasingly used to develop complex applications. As OOP 
10 moves toward the mainstream of software design and development, various software 
solutions require adaptation to make use of the benefits of OOP, A need exists for 

13 these principles of OOP to be applied to a messaging interface of an electronic 

messaging system such that a set of OOP classes and objects for the messaging 

fz interface can be provided. 

m 15 

'[q OOP is a process of developing computer software using objects, including the steps 

of analyzing the problem, designing the system, and constructing the program. An 
IB object is a software package that contains both data and a collection of related 

i IS 

fll structures and procedures. Since it contains both data and a collection of structures 

20 and procedures, it can be visualized as a self-sufficient component that does not 
require other additional structures, procedures or data to perform its specific task. 
OOP, therefore, views a computer program as a collection of largely autonomous 
components, called objects, each of which is responsible for a specific task. This 
concept of packaging data, structures, and procedures together in one component or 
25 module is called encapsulation. 

In general, OOP components are reusable software modules which present an 
interface that conforms to an object model and which are accessed at run-time 
through a component integration architecture. A component integration architecture 
30 is a set of architecture mechanisms which allow software modules in different 

process spaces to utilize each others capabilities or fimctions. This is generally done 
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by assuming a common component object model on which to build the architecture. 
It is worthwhile to differentiate between an object and a class of objects at this point. 
An object is a single instance of the class of objects, which is often just called a 
class. A class of objects can be viewed as a blueprint, from which many objects can 
5 be formed. 

OOP allows the programmer to create an object that is a part of another object. For 
example, the object representing a piston engine is said to have a composition- 
relationship with the object representing a piston. In reality, a piston engine 
10 comprises a piston, valves and many other components; the fact that a piston is an 
element of a piston engine can be logically and semantically represented in OOP by 
% two objects. 

m OOP also allows creation of an object that "depends from" another object. If there 

15 are two objects, one representing a piston engine and the other representing a piston 
engine wherein the piston is made of ceramic, then the relationship between the two 
f% objects is not that of composition. A ceramic piston engine does not make up a 

Sf^J piston engine. Rather it is merely one kind of piston engine that has one more 

111 limitation than the piston engine; its piston is made of ceramic. In this case, the 

20 object representing the ceramic piston engine is called a derived object, and it 
inherits all of the aspects of the object representing the piston engine and adds 
further Umitation or detail to it. The object representing the ceramic piston engine 
"depends from" the object representing the piston engine. The relationship between 
these objects is called inheritance. 

25 

When the object or class representing the ceramic piston engine inherits all of the 
aspects of the objects representing the piston engine, it inherits the thermal 
characteristics of a standard piston defined in the piston engine class. However, the 
ceramic piston engine object overrides these ceramic specific thermal characteristics, 
30 which are typically different from those associated with a metal piston. It skips over 
the original and uses new functions related to ceramic pistons. Different kinds of 
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piston engines have different characteristics, but may have the same underlying 
functions associated with it (e.g., how many pistons in the engine, ignition 
sequences, lubrication, etc.). To access each of these functions in any piston engine 
object, a programmer would call the same functions with the same names, but each 
5 type of piston engine may have different/overriding implementations of functions 
behind the same name. This ability to hide different implementations of a function 
behind the same name is called polymorphism and it greatly simplifies 
communication among objects. 

10 With the concepts of composition-relationship, encapsulation, inheritance and 

polymorphism, an object can represent just about anything in the real world. In fact, 
our logical perception of the reality is the only limit on determining the kinds of 
things that can become objects in object-oriented software. Some typical categories 
are as follows: 

15 • Objects can represent physical objects, such as automobiles in a traffic-flow 
simulation, electrical components in a circuit-design program, countries in an 
economics model, or aircraft in an air-traffic-control system. 

• Objects can represent elements of the computer-user envirormient such as 
windows, menus or graphics objects. 

20 • An object can represent an inventory, such as a persormel file or a table of 
the latitudes and longitudes of cities. 

• An object can represent user-defined data types such as time, angles, and 
complex numbers, or points on the plane. 

25 With this enormous capability of an object to represent just about any logically 
separable matters, OOP allows the software developer to design and implement a 
computer program that is a model of some aspects of reality, whether that reality is a 
physical entity, a process, a system, or a composition of matter. Since the object can 
represent anything, the software developer can create an object which can be used as 

30 a component in a larger software project in the future. 
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If 90% of a new OOP software program consists of proven, existing components 
made from preexisting reusable objects, then only the remaining 10% of the new 
software project has to be written and tested from scratch. Since 90% already came 
from an inventory of extensively tested reusable objects, the potential domain from 
which an error could originate is 10% of the program. As a result, OOP enables 
software developers to build objects out of other, previously built objects. 

This process closely resembles complex machinery being built out of assembhes and 
sub-assembUes. OOP technology, therefore, makes software engineering more like 
hardware engineering in that software is built from existing components, which are 
available to the developer as objects. All this adds up to an improved quality of the 
software as well as an increased speed of its development. 

Programming languages are beginning to fially support the OOP principles, such as 
encapsulation, inheritance, polymorphism, and composition-relationship. With the 
advent of the C+-h language, many commercial software developers have embraced 
OOP. C++ is an OOP language that offers a fast, machine-executable code. 
Furthermore, C+-h is suitable for both commercial-apphcation and systems- 
programming projects. For now, C++ appears to be the most popular choice among 
many OOP programmers, but there is a host of other OOP languages, such as 
Smalltalk, Common Lisp Object System (CLOS), and Eiffel. Additionally, OOP 
capabiUties are being added to more traditional popular computer programming 
languages such as Pascal. 

The benefits of object classes can be summarized, as follows: 

• Objects and their corresponding classes break down complex programming 
problems into many smaller, simpler problems. 

• Encapsulation enforces data abstraction through the organization of data into 
small, independent objects that can communicate with each other. 
Encapsulation protects the data in an object from accidental damage, but 
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allows other objects to interact with that data by calling the object's member 
functions and structures. 

Subclassing and inheritance make it possible to extend and modify objects 
through deriving new kinds of objects from the standard classes available in 
the system. Thus, new capabilities are created without having to start from 
scratch. 

Polymorphism and multiple inheritance make it possible for different 
programmers to mix and match characteristics of many different classes and 
create specialized objects that can still work with related objects in 
predictable ways. 

Class hierarchies and containment hierarchies provide a flexible mechanism 
for modeling real-world objects and the relationships among them. 
Libraries of reusable classes are useful in many situations, but they also have 
some limitations. For example; 

Complexity. In a complex system, the class hierarchies for related classes 
can become extremely confusing, with many dozens or even hundreds of 
classes. 

Flow of control. A program written with the aid of class libraries is still 
responsible for the flow of control (i.e., it must control the interactions 
among all the objects created from a particular library). The programmer has 
to decide which functions to call at what times for which kinds of objects. 
Duplication of effort. Although class libraries allow programmers to use and 
reuse many small pieces of code, each programmer puts those pieces together 
in a different way. Two different programmers can use the same set of class 
libraries to write two programs that do exactly the same thing but whose 
intemal structure (i.e., design) may be quite different, depending on hundreds 
of small decisions each programmer makes along the way. Inevitably, 
similar pieces of code end up doing similar things in slightly different ways 
and do not work as well together as they should. 
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Class libraries are very flexible. As programs grow more complex, more 
programmers are forced to reinvent basic solutions to basic problems over and over 
again. A relatively new extension of the class library concept is to have a 
framework of class libraries. This framework is more complex and consists of 
5 significant collections of collaborating classes that capture both the small scale 
patterns and major mechanisms that implement the common requirements and 
design in a specific application domain. They were first developed to fi'ee 
application programmers from the chores involved in displaying menus, windows, 
dialog boxes, and other standard user interface elements for personal computers. 

10 

Frameworks also represent a change in the way programmers think about the 
43 interaction between the code they write and code written by others. In the early days 

of procedural programming, the programmer called libraries provided by the 
operating system to perform certain tasks, but basically the program executed down 
15 the page from start to finish, and the programmer was solely responsible for the flow 
of control. This was appropriate for printing out paychecks, calculating a 
mathematical table, or solving other problems with a program that executed in just 
nj one way, 

20 The development of graphical user interfaces began to turn this procedural 

programming arrangement inside out. These interfaces allow the user, rather than 
program logic, to drive the program and decide when certain actions should be 
performed. Today, most personal computer software accomplishes this by means of 
an event loop which monitors the mouse, keyboard, and other sources of external 

25 events and calls the appropriate parts of the programmer's code according to actions 
that the user performs. The programmer no longer determines the order in which 
events occur. Instead, a program is divided into separate pieces that are called at 
unpredictable times and in an unpredictable order. By relinquishing control in this 
way to users, the developer creates a program that is much easier to use. 

30 Nevertheless, individual pieces of the program written by the developer still call 
libraries provided by the operating system to accomplish certain tasks, and the 



-18- 



programmer must still determine the flow of control within each piece after it's 
called by the event loop. Application code still "sits on top of the system. 

Even event loop programs require programmers to write a lot of code that should not 
5 need to be written separately for every application. The concept of an application 
framework carries the event loop concept further. Instead of dealing with all the 
nuts and bolts of constructing basic menus, windows, and dialog boxes and then 
making these things all work together, programmers using application frameworks 
start with working application code and basic user interface elements in place. 
10 Subsequently, they build from there by replacing some of the generic capabiUties of 
the framework with the specific capabilities of the intended appUcation. 

Apphcation frameworks reduce the total amount of code that a programmer has to 
write from scratch. However, because the framework is really a generic application 
15 that displays windows, supports copy and paste, and so on, the programmer can also 
relinquish control to a greater degree than event loop programs permit. The 
framework code takes care of almost all event handling and flow of control, and the 
programmer's code is called only when the framework needs it (e.g., to create or 
manipulate a proprietary data structure). 

20 

A programmer writing a framework program not only relinquishes control to the 
user (as is also true for event loop programs), but also relinquishes the detailed flow 
of control within the program to the framework. This approach allows the creation 
of more complex systems that work together in interesting ways, as opposed to 
25 isolated programs, having custom code, being created over and over again for 
similar problems. 

Thus, as is explained above, a framework basically is a collection of cooperating 
classes that make up a reusable design solution for a given problem domain. It 
30 typically includes objects that provide default behavior (e.g., for menus and 

windows), and programmers use it by inheriting some of that default behavior and 
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overriding other behavior so that the framework calls application code at the 
appropriate times. 

There are three main differences between frameworks and class libraries: 
5 • Behavior versus protocol Class libraries are essentially collections of 

behaviors that you can call when you want those individual behaviors in your 
program. A framework, on the other hand, provides not only behavior but 
also the protocol or set of rules that govern the ways in which behaviors can 
be combined, including rules for what a programmer is supposed to provide 

10 versus what the framework provides. 

• Call versus override. With a class library, the code the programmer 
instantiates objects and calls their member fixnctions. It's possible to 
instantiate and call objects in the same way with a framework (i.e., to treat 
the framework as a class library), but to take fiiU advantage of a framework's 

1 5 reusable design, a programmer typically writes code that overrides and is 

called by the framework. The framework manages the flow of control 
among its objects. Writing a program involves dividing responsibiUties 
among the various pieces of software that are called by the framework rather 
than specifying how the different pieces should work together. 

20 • Implementation versus design. With class libraries, programmers reuse only 
implementations, whereas with frameworks, they reuse design. A framework 
embodies the way a family of related programs or pieces of software work. 
It represents a generic design solution that can be adapted to a variety of 
specific problems in a given domain. For example, a single framework can 

25 embody the way a user interface works, even though two different user 

interfaces created with the same framework might solve quite different 
interface problems. 

Thus, through the development of frameworks for solutions to various problems and 
30 programming tasks, significant reductions in the design and development effort for 
software can be achieved. A preferred embodiment of the invention utilizes 
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HyperText Markup Language (HTML) to implement documents on the Internet 
together with a general-purpose secure communication protocol for a transport 
medium between the client and the Newco. HTTP or other protocols could be 
readily substituted for HTML without undue experimentation. Information on these 
products is available in T. Bemers-Lee, D, Connoly, "RFC 1866: Hypertext Markup 
Language - 2.0" (Nov. 1995); and R. Fielding, H, Frystyk, T. Bemers-Lee, J. Gettys 
and J.C. Mogul, "Hypertext Transfer Protocol HTTP/1 . 1 : HTTP Working Group 
Intemet Draft" (May 2, 1996). HTML is a simple data format used to create 
hypertext documents that are portable from one platform to another. HTML 
documents are SGML documents with generic semantics that are appropriate for 
representing information from a wide range of domains. HTML has been in use by 
the World-Wide Web global information initiative since 1990. HTML is an 
application of ISO Standard 8879; 1986 Information Processing Text and Office 
Systems; Standard GeneraUzed Markup Language (SGML). 

To date, Web development tools have been limited in their ability to create dynamic 
Web applications which span from client to server and interoperate with existing 
computing resources. Until recently, HTML has been the dominant technology used 
in development of Web-based solutions. However, HTML has proven to be 
inadequate in the following areas: 

• Poor performance; 

• Restricted user interface capabilities; 

• Can only produce static Web pages; 

• Lack of interoperability with existing applications and data; and 

• Inability to scale. 

Sun Microsystem's Java language solves many of the client-side problems by: 

• Improving performance on the cUent side; 

• EnabUng the creation of dynamic, real-time Web appUcations; and 

• Providing the ability to create a wide variety of user interface components. 
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With Java, developers can create robust User Interface (UI) components. Custom 
"widgets" (e.g., real-time stock tickers, animated icons, etc.) can be created, and 
client-side performance is improved. Unlike HTML, Java supports the notion of 
client-side validation, offloading appropriate processing onto the client for improved 
5 performance. Dynamic, real-time Web pages can be created. Using the above- 
mentioned custom UI components, dynamic Web pages can also be created. 

Sun's Java language has emerged as an industry-recognized language for 
"programming the Internet." Sun defines Java as: "a simple, object-oriented, 
distributed, interpreted, robust, secure, architecture-neutral, portable, high- 
performance, multithreaded, dynamic, buzzword-compHant, general-purpose 
programming language. Java supports programming for the Internet in the form of 
platform-independent Java applets." Java applets are small, speciaUzed appUcations 
that comply with Sun's Java AppUcation Programming Interface (API) allowing 
developers to add "interactive content" to Web documents (e.g., simple animations, 
page adornments, basic games, etc.). Applets execute within a Java-compatible 
browser (e.g., Netscape Navigator) by copying code from the server to client. From 
a language standpoint, Java's core feature set is based on C++. Sun's Java hterature 
states that Java is basically, "C++ with extensions from Objective C for more 
dynamic method resolution." 

Another technology that provides similar fimction to JAVA is provided by Microsoft 
and ActiveX Technologies, to give developers and Web designers wherewithal to 
build dynamic content for the Intemet and personal computers. ActiveX includes tools 
25 for developing animation, 3-D virtual reality, video and other multimedia content. The 
tools use hatemet standards, work on multiple platforms, and are being supported by 
over 100 companies. The group's building blocks are called ActiveX Controls, small, 
fast components that enable developers to embed parts of software in hypertext 
markup language (HTML) pages. ActiveX Controls work with a variety of 
30 programming languages including Microsoft Visual C++, Borland Delphi, Microsoft 
Visual Basic programming system and, in the fixture, Microsoft's development tool for 
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Java, code named "Jakarta." ActiveX Technologies also includes ActiveX Server 
Framework, allowing developers to create server applications. One of ordinary skill in 
the art readily recognizes that ActiveX could be substituted for JAVA without undue 
experimentation to practice the invention, 

5 

A simulation engine in accordance with a preferred embodiment is based on a 
Microsoft Visual Basic component developed to help design and test feedback in 
relation to a Microsoft Excel spreadsheet. These spreadsheet models are what 
simulate actual business fimctions and become a task that will be performed by a 
10 student The Simulation Engine accepts simulation inputs and calculates various 
outputs and notifies the system of the status of the simulation at a given time in 
order to obtain appropriate feedback. 

Relationship of Components 

The simulation model executes the business fimction that the student is learning and 
15 is therefore the center point of the application. An activity * layer' allows the user to 
visually guide the simulation by passing inputs into the simulation engine and 
receiving an output from the simulation model. For example, if the student was 
working on an income statement activity, the net sales and cost of goods sold 
calculations are passed as inputs to the simulation model and the net income value is 
20 calculated and retrieved as an output. As calculations are passed to and retrieved 
from the simulation model, they are also passed to the Intelligent Coaching Agent 
(ICA). The ICA analyzes the Inputs and Outputs to the simulation model and 
generates feedback based on a set of rules. This feedback is received and displayed 
through the Visual Basic Architecture. 

25 

Figure 2 is a block diagram of a system architecture in accordance with a preferred 
embodiment. The Presentation 'layer' 210 is separate from the activity 'layer' 220 
and communication is facihtated through a set of messages 230 that control the 
display specific content topics. A preferred embodiment enables knowledge workers 
30 200 & 201 to acquire complex skills rapidly, reliably and consistently across an 
organization to deliver rapid acquisition of complex skills. This result is achieved 
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by placing individuals in a simulated business environment that "looks and feels" 
like real work, and challenging them to make decisions which support a business' 
strategic objectives utilizing highly effective learning theory (e.g., goal based 
learning, learn by doing, failure based learning, etc.), and the latest in multimedia 
5 user interfaces, coupled with three powerful, integrated software components. The 
first of these components is a software Solution Construction Aid (SCA) 230 
consisting of a mathematical modeling tool 234 which simulates business outcomes 
of an individual's collective actions over a period of time. The second component is 
a knowledge system 250 consisting of an HTML content layer which organizes and 
10 presents packaged knowledge much like an online text book with practice exercises, 
video war stories, and a glossary. The third component is a software tutor 270 

^; comprising an artificial intelligence engine 240 which generates individualized 

0 coaching messages based on decisions made by leamer. 

^ 15 Feedback is unique for each individual completing the course and supports chent 
S cultural messages 242 "designed into" the course. A business simulation 

methodology that includes support for content acquisition, story line design, 
S interaction design, feedback and coaching deUvery, and content delivery is 

ll architected into the system in accordance with a preferred embodiment. A large 

J 20 number of "pre-designed" learning interactions such as drag and drop association of 
information 238, situation assessment/action plaiming, interviewing (one-on-one, 
one-to-many), presenting (to a group of experts/executives), metering of 
performance (handle now, handle later), "time jumping" for impact of decisions, 
competitive landscape shift (while "time jumping", competitors merge, customers 
25 are acquired, etc.) and video interviewing with automated note taking are also 
included in accordance with a preferred embodiment. 

Business simulation in accordance with a preferred embodiment delivers training 
curricula in an optimal manner. This is because such apphcations provide effective 
30 training that mirrors a student's actual work environment. The application of skills 
"on the job" facilitates increased retention and higher overall job performance. 



-24- 



While the results of such training appUcations are impressive, business simulations 
are very complex to design and build correctly. These simulations are characterized 
by a very open-ended environment, where students can go through the application 
along any number of paths, depending on their learning style and prior 
5 experiences/knowledge. 

A category of learning approaches called Learn by Doing, is commonly used as a 
solution to support the first phase (Leam) of the Workforce Performance Cycle, 
However, it can also be a solution to support the second phase (Perform) of the cycle 
10 to enable point of need learning during job performance. By adopting the approach 
presented, some of the benefits of a technology based approach for building business 
15 simulation solutions which create more repeatable, predictable projects resulting in 

more perceived and actual user value at a lower cost and in less time are highlighted. 

PI 15 Most corporate training programs today are misdirected because they have failed to 
[1 focus properly on the purpose of their training. These programs have confiised the 

^' memorization of facts with the ability to perform tasks; the knowing of "that" with 

(3 the knowing of "how". By adopting the methods of traditional schools, businesses 

:M are teaching a wide breadth of disconnected, decontextualized facts and figures, 

0 20 when they should be focused on improved performance. How do you teach 

performance, when lectures, books, and tests inherently are designed around facts 
and figures? Throw away the lectures, books, and tests. The best way to prepare for 
high performance is to perform; experience is the best teacher! Most business 
leaders agree that workers become more effective the more time they spend in their 
25 jobs. The best approach for training novice employees, therefore, would be letting 
them leam on the job, acquiring skills in their actual work environment. The idea of 
leaming-by-doing is not revolutionary, yet it is resisted in business and academia. 
Why is this so, if higher competence is universally desired? 

30 Learners are reluctant to adopt leaming-by-doing because they are frightened of 
failure. People work hard to avoid making mistakes in front of others. Business 
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leaders are hesitant to implement leaming-by-doing because novice failure may have 
dramatic safety, legal and financial implications. Imagine a novice pilot leaming- 
by-doing as he accelerates a large jet plane down a runway; likewise, consider a new 
financial analyst leaming-by-doing as he structures a multi-million dollar financial 
5 loan. Few employers are willing to endure such failures to have a more competent 
workforce. 

The concerns of employee and employer can be relieved if the training (and 
accompanying failure) didn't occur in firont of co-workers and clients, if it didn't 

10 jeopardize a multi-million dollar aircraft or a multi-million dollar deal What if the 
training was performed privately, in a richly modeled simulation of the workers 
actual job? In a simulated environment, failure would result in dedicated instruction 
instead of embarrassment, injury, or monetary losses. Simulated environments 
provide a sense of liberation for exploration that does not exist in the real world. 

15 Knowing that the consequences of experimentation will unlikely be dire, leamers are 
able to explore the "what if..." inherent in us all. In this way, the best way to prepare 
for high performance is to simulate actual performance. New media technologies 
utiUzing multimedia provide the possibiHty to create such business simulation 
experiences. 

20 

Even if companies didn't make the mistake of focusing on "what" learning vs. "how" 
learning, they still tend to have the overly narrow view of education/training as 
something that only occurs prior to workers being asked to actually perform their 
job. Learning is something that is constantly occurring, and doesn't cease once "real 

25 work" has begun. The closer new lessons occur in time with the application of those 
lessons, the greater the resultant learning. This fact helps to explain some of the 
reasoning behind the benefits of business simulation, described in the previous 
section. In those systems, all new lessons are taught in close relationship to their 
real world use; everything is in context and, most importantly, are presented at the 

30 appropriate time. But as the properly trained worker performs their job, the working 
environment changes. New demands occur, and new methods and rules are 
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developed. As these events occur, there is a need for new support/training that, in 
most cases, must wait to be addressed until the next "pre-performance" training 
session. 

5 In these cases, the need (or demand) for additional training doesn't match the supply. 
This lag between a training need and the fulfilUng lesson has a dramatic negative 
impact on productivity, accuracy, and customer satisfaction. Workers need to have 
the opportunity to leam while they are performing. Just as during pre-performance 
training, one powerful mechanism for identifying and correcting (simulated) 

10 performance problems is to have an expert available at all time to watch your actions 
and remediate when appropriate. This, obviously, is too costly and time intensive of 
an approach to be practical with actual experts. But what if workers had access to a 
support system that provided the majority of the benefits of a real expert, 
transparently integrated into their work environment? Such a system would provide 

1 5 advice at key moments in the work flow for problem resolution and/or process 
improvement, tools to ease task completion, reference material of best practice 
knowledge, and point of need training courses. With a support system that 
proactively assists the worker in performance of their job tasks at a higher level of 
competency, productivity and customer satisfaction (both internal and external) 

20 would soar. 

The key to such a support system is that it is seamlessly integrated into the business 
system that the knowledge worker uses to execute their job tasks. Workers don't 
need to go "off-line" or seek out cryptic information buried within paper manuals 
25 and binders for guidance or to find the answer to queries. All the support 

components are made available through the same applications the worker's use, at 
the point in which they need them, tailored to the individual to show "how", not just 
"what". Leaming would be occurring all the time, with little distinction between 
performing and improving performance. 

30 
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Establishing that training should focus on performance (how), rather than facts 
(what), and extending the model of learning to include assistance while performing, 
rather than only before performance, still leaves us dangerously exposed in 
preparing to compete in the new, chaotic economy. As was mentioned in the 
5 opening of this paper, the pace of change in business today is whiplash fast. Not 
only are new methods of doing business evolving every 18-24 months, new 
competitors emerge, dominate, and fade in time periods businesses used to take to 
perform demographic studies. Now more than ever, those who do not reinvent 
themselves on a regular basis will be fossilized by the pace of change. 

10 

Even the best pre-performance training and the most advanced performance support 
tools are destined to be outdated if there isn't a fresh supply of real-world 
requirements and lessons leamed being fed back as inputs for the next go 'round. 
Innovation is a key step in the Workforce Performance Cycle. This step requires 
15 business to employ Stephen Covers famous habit of "sharpening the saw", or "take 
time to be fast". 

There is an untold wealth of information buried within the heads of business users 
responsible for implementing the steps outlined in their pre-performance training 

20 and performance support tools. No other group within an organization can more 
accurately assess the effectiveness of current methods, or project needed changes 
that will have dramatic future impact. This step of reflecting on the current and past 
state of affairs, uncovering new approaches by identifying what is working and 
what is not, and adapting accordingly for the future is the last phase of the 

25 learning/performance cycle. 

Innovation to fuel future training and performance support comes directly from the 
community most closely tied to task performance. Effective businesses need to 
develop and cultivate a mechanism for communication and collaboration among the 
30 experts in these communities to more efficiently benefit from their collective 
wisdom. In today's business, that which is evident to your business is evident to 
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nearly all your competitors as well. The competitive advantage lies in uncovering 
that which is unexpected or not immediately apparent, adapting your business 
processes to exploit the discovery, and quickly, but effectively, educating your 
workforce on the new policies and procedures, all before the competition catches on 
5 or the market changes again. 

This innovation process is the critical final step in continuous education of the most 
effective and up-to-date policies, procedures, and information. Without formaUzed 
innovation, companies not only risk being a step behind the ever advancing 

1 0 competition, but compound the problem by continuing to train their personnel with 
outdated strategies and information. One way to formalize this vital step in the 
Workforce Performance Cycle is to construct Virtual Leaming Communities, where 
many 'experts' can share experiences, submit ideas for improvements, play out 
* Vhat-if scenarios, and contribute on complex problems that may be 

15 insurmountable without significant collaboration with others. Such Leaming 
Conmiunities could nurture up-to-date discussion of what is actually happening 
within a business, eliminating the traditional information-passing lag that plagues 
many business as new data travels through corporate hierarchies. This increased 
nimbleness would help organizations quickly address new competitive trends and 

20 outdated strategies, identify potential solutions, and implement improved processes 
in the form of updated training and performance support reference materials. 

Currently, a typical BusSim engagement takes between one and two years to 
complete and requires a variety of both functional and technical skills. Figure 3 

25 depicts the timeline and relative resource requirements for each phase of 

development for a typical appUcation development in accordance with a preferred 
embodiment. The chart clearly depicts the relationship between the large number of 
technical resources required for both the build aad test phases of development. This 
is because the traditional development process used to build BusSim solutions 

30 reflects more of a "one off philosophy, where development is done from scratch in 
a monolithic fashion, with little or no reuse firom one application to the next. This 
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lack of reuse makes this approach prohibitively expensive, as well as lengthy, for 
fiiture BusSim projects. 

The solution to this problem is to put tools in the hands of instructional designers 
that allows them to create their BusSim designs and implement them without the 
need for programmers to write code. And to put application architectures that 
integrate with the tools in the hands of developers, providing them with the ability to 
quickly dehver solutions for a number of different platforms. The reuse, then, comes 
in using the tools and architectures from one engagement to another. Both 
functional and technical resources carry with them the knowledge of how to use the 
technology, which also has an associated benefit of establishing a best-practice 
development methodology for BusSim engagements. 

The tools and architectures described herein are part of a next-generation Business 
Simulation delivery framework that will reduce the total effort necessary to build 
solutions in accordance with a preferred embodiment. Figure 4 depicts the potential 
savings in both functional and technical tasks in accordance with a preferred 
embodiment. 

Development Cycle Activities 

Design Phase 

In the Design Phase, instructional designers become oriented to the content area and 
begin to conceptualize an instructional approach. They famiUarize themselves with 
the subject matter through reading materials and interviews with Subject Matter 
Experts (SMEs). They also identify learning objectives from key client contacts. 
Conceptual designs for student interactions and interface layouts also begin to 
emerge. After the conceptual designs have taken shape, Low-Fi user testing (a.k.a. 
Conference Room Piloting) is performed. Students interact with interface mock-ups 
while facilitators observe and record any issues. Finally, detailed designs are created 
that incorporate findings. These detailed designs are handed off to the development 
team for implementation. 
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The design phase has traditionally been fraught with several problems. Unlike a 
traditional business system, BusSim solutions are not rooted in tangible business 
processes, so requirements are difficult to identify in a concrete way. This leaves 
5 instructional designers with a 'blue sky' design problem. With few business-driven 
constraints on the solution, shallow expertise in the content area, and Umited 
technical skills, instructional designers have little help in beginning a design. 
Typically, only experienced designers have been able to conjure interface, analysis, 
and feedback designs that meet the learning objectives yet remain technically 
10 feasible to implement. To compound the problem, BusSim solutions are very open 
ended in nature. The designer must anticipate a huge combination of student 
behavior to design feedback that is helpfiil and reahstic. 

Build Phase 

1 5 During the build phase, the apphcation development team uses the detailed designs 
to code the application. Coding tasks include the interfaces and widgets that the 
student interacts with. The interfaces can be made up of buttons, grids, check boxes, 
or any other screen controls that allow the student to view and manipulate his 
deUverables. The developer must also code logic that analyzes the student's work 

20 and provides feedback interactions. These interactions may take the form of text 
and/or multimedia feedback from simulated team members, conversations with 
simulated team members, or direct manipulations of the student's work by 
simulated team members. In parallel with these coding efforts, graphics, videos, and 
audio are being created for use in the application. Managing the development of 

25 these assets have their own complications. 

Risks in the build phase include misinterpretation of the designs. If the developer 
does not accurately understand the designer's intentions, the application will not 
fimction as desired. Also, coding these appUcations requires very skilled developers 
30 because the logic that analyzes the student's work and composes feedback is very 
complex. 
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Test Phase 

The Test Phase, as the name implies, is for testing the appHcation. Testing is 
performed to verify the appUcation in three ways: first that the appUcation functions 
5 properly (functional testing), second that the students understand the interface and 
can navigate effectively (usabiUty testing), and third that the learning objectives are 
met (cognition testing). Functional testing of the application can be carried out by 
the development team or by a dedicated test team. If the application fails to function 
properly, it is debugged, fixed, recompiled and retested until its operation is 
10 satisfactory. Usability and cognition testing can only be carried out by test students 
who are unfamiliar with the application. If usability is unsatisfactory, parts of the 
interface and or feedback logic may need to be redesigned, recoded, and retested. If 
the learning objectives are not met, large parts of the application may need to be 
removed and completely redeveloped fi"om a different perspective. 

15 

The test phase is typically where most of the difficulties in the BusSim development 
cycle are encountered. The process of discovering and fixing functional, usability, 
and cognition problems is a difficult process and not an exact science. 

20 For functional testing, testers operate the application, either by following a test script 
or by acting spontaneously and documenting their actions as they go. When a 
problem or unexpected result is encountered, it too is documented. The application 
developer responsible for that part of the appHcation then receives the 
documentation and attempts to duplicate the problem by repeating the tester's 

25 actions. When the problem is duplicated, the developer investigates further to find 
the cause and implement a fix. The developer once again repeats the tester's actions 
to verify that the fix solved the problem. Finally, all other test scripts must be rerun 
to verify that the fix did not have unintended consequences elsewhere in the 
application. 



30 
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This process has inherent difficulty. If the tester is inaccurate in recording his 
actions, or the developer is inaccurate in repeating them, then the problem may never 
be duplicated and the defect never found. Furthermore, the problem may be 
dependent on some condition in the tester's environment that is not readily 
5 observable or is not even related to the application. This process has proven to be 
tedious, time-consuming, and of limited reliabihty. 

For usability testing, test students operate the application as it will be operated in 
production. Ideally, their progress is only impeded by their lack of mastery of the 
10 content. As they gain proficiency, they are able to complete the activities and move 
on. As is often the case, however, they are impeded by unclear instructions, a non- 
intuitive interface, or other usability shortcomings. When these difficulties are 

m encountered, the facilitators document the problems and student comments on what 

j^i is needed to improve usability, 

ill 15 

kQ There are two major risks associated with usability testing. First, student action 

]^ recording is not as rigorous because actual students are performing the testing, so 

fimctional problems that don't appear until now are difficult to reproduce. Second, 
ill resolutions to usability problems often involve significant modification to the 

20 application code and interface which requires repeating of portions of design, build, 
and test. 

For cognition testing, students are surveyed and/or tested to determine their level of 
understanding of the material. If results indicate that the learning objectives are not 
25 being adequately met, the design is reevaluated. Changes proposed to improve the 
cognition may include massive redesign and rebuilding. 

Execution Phase 

The Execution Phase refers to the steady state operation of the completed application 
30 in its production environment. For some chents, this involves phone support for 
students. Clients may also want the ability to track students' progress and control 
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their progression through the course. Lastly, clients may want the ability to track 
issues so they may be considered for inclusion in course maintenance releases. 

One of the key values of on-line courses is that they can be taken at a time, location, 
5 and pace that is convenient for the individual student. However, because students 
are not centrally located, support is not always readily available. For this reason it is 
often desirable to have phone support for students. 



CUents may also desire to track students' progress, or control their advancement 
10 through the course. Under this strategy, after a student completes a section of the 
course, he will transfer his progress data to a processing center either electronically 
or by physically mailing a disk. There it can be analyzed to verify that he completed 
all required work satisfactorily. One difficulty commonly associated with student 
tracking is isolating the student data for analysis. It can be unwieldy to transmit all 
15 the course data, so it is often imperative to isolate the minimum data required to 
perform the necessary analysis of the student's progress. 

A Delivery Framework for Business Simulation 

As discussed earlier, the traditional development process used to build BusSim 
20 solutions reflects more of a "one off philosophy, where development is done from 
scratch in a monolithic fashion, with little or no reuse from one application to the 
next. A better approach would be to focus on reducing the total effort required for 
development through reuse, which, in tum would decrease cost and development 
time. 

25 

The first step in considering reuse as an option is the identification of common 
aspects of the different BusSim applications that can be generalized to be usefiil in 
fixture applications. In examination of the elements that make up these apphcations, 
three common aspects emerge as integral parts of each: 
30 ■ Interface 
■ Analysis 
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■ Interpretation 
Interface 

Every BusSim application must have a mechanism for interaction with the student. 
5 The degree of complexity of each interface may vary, from the high interactivity of a 
high-fidelity real-time simulation task, to the less complex information delivery 
requirements of a business case background information task. Regardless of how 
sophisticated the User Interface (UI), it is a vital piece of making the underlying 
simulation and feedback logic useful to the end user. 

10 

Analysis 

Every BusSim appHcation does analysis on the data that defines the current state of 
the simulation many times throughout the execution of the application. This 
analysis is done either to determine what is happening in the simulation, or to 
1 5 perform additional calculations on the data which are then fed back into the 

simulation. For example, the analysis may be the recognition of any actions the 
student has taken on artifacts within the simulated environment (notebooks, number 
values, interviews conducted, etc.), or it may be the calculation of an ROI based on 
numbers the student has supplied. 

20 

Interpretation 

Substantive, useful feedback is a critical piece of any BusSim application. It is the 
main mechanism to communicate if actions taken by the student are helping or 
hurting them meet their performance objectives. The interpretation piece of the set 

25 of proposed commonalties takes the results of any analysis performed and makes 
sense of it. It takes the non-biased view of the world that the Analysis portion 
delivers (i.e., "Demand is up 3%") and places some evaluative context around it (i.e., 
"Demand is below the expected 7%; youYe in trouble!", or "Demand has exceeded 
projections of 1.5%; Great job!"). Figure 5 illustrates commonalties in accordance 

30 with a preferred embodiment. 
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Common Features of Business Simulation Applications 

There are several approaches to capturing commonalties for reuse. Two of the more 
common approaches are framework-based and component-based. To help illustrate 
the differences between the two approaches, we will draw an analogy between 
5 building an appUcation and building a house. One can construct a house from 
scratch, using the raw materials, 2x4s, nails, paint, concrete, etc. One can also 
construct an application from scratch, using the raw materials of new designs and 
new code. The effort involved in both undertakings can be reduced through 
framework-based and/or component-based reuse. 

10 

Framework-Based Reuse 

Within the paradigm of framework-based reuse, a generic framework or architecture 
M is constructed that contains commonalties. In the house analogy, one could purchase 

in a prefabricated house framework consisting of floors, outside walls, bearing walls 

^ 15 and a roof The house can be customized by adding partition walls, wall-paper, 
woodwork, carpeting etc. Similarly, prefabricated appUcation frameworks are 
"f^ available that contain basehne appUcation structure and ftmctionaUty. Individual 

® applications are conipleted by adding specific functionality and customizing the 

jl look-and-feel. An example of a commonly used ^pUcation framework is Microsoft 

Tl 20 Foundation Classes. It is a framework for developing Windows appUcations using 
C-i-+. MFC supplies the base functionality of a windowing appUcation and the 
developer completes the application by adding functionality within the framework. 
Framework-based reuse is best suited for capturing template-like features, for 
example user interface management, procedural object behaviors, and any other 
25 features that may require specialization. 

Some benefits of using a framework include: 

■ Extensive functionality can be incorporated into a framework. In the house 
analogy, if I know I am going to build a whole neighborhood of three bedroom 
30 ranches, I can build the plumbing, wiring, and partition walls right into the 

framework, reducing the incremental effort required for each house. If I know I am 
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going to build a large number of very similar applications, they will have more 
commonalties that can be included in the framework rather than built individually. 

■ Applications can override the framework-supplied functionality wherever 
appropriate. If a house framework came with pre-painted walls, the builder could 

5 just paint over them with preferred colors. Similarly, the object oriented principle of 
inheritance allows an application developer to override the behavior of the 
framework. 

Component-Based Reuse 

In the paradigm of component-based reuse, key functionahty is encapsulated in a 
component. The component can then be reused in multiple appUcations, In the 
house analogy, components correspond to apphances such as dishwashers, 
refrigerators, microwaves, etc. 

Similarly, many application components with pre-packaged functionality are 
available from a variety of vendors. An example of a popular component is a Data 
Grid. It is a component that can be integrated into an application to deliver the 
capability of viewing columnar data in a spreadsheet-like grid. Component-based 
reuse is best suited for capturing black-box-like features, for example text 
processing, data manipulation, or any other features that do not require 
specialization. 

Some benefits of using components include: 

■ Several applications on the same computer can share a single component 

This is not such a good fit with the analogy, but imagine if all the houses in a 
25 neighborhood could share the same dishwasher simultaneously. Each home would 
have to supply its own dishes, detergent, and water, but they could all wash dishes 
in parallel. In the application component world, this type of sharing is easily 
accomplished and results in reduced disk and memory requirements. 

■ Components tend to be less platform and tool dependent. A microwave can 
30 be used in virtually any house, whether it's framework is steel or wood, and 

regardless of whether it was customized for building mansions or shacks. You can 
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put a high-end microwave in a low-end house and vice- versa. You can even have 
multiple different microwaves in your house. Component technologies such as 
CORBA, COM, and Java Beans make this kind of flexibility conrmionplace in 
application development. 

The Solution: A Combined Approach 

Often, the best answer to achieving reuse is through a combination of framework- 
based and component-based techniques. A framework-based approach for building 
BusSim apphcations is appropriate for developing the user interface, handling user 
and system events, starting and stopping the application, and other appHcation- 
specific and delivery platform-specific fimctions. A component-based approach is 
appropriate for black-box functionality. That is, fimctionahty that can be used as-is 
with no specialization required. 

In creating architectures to support BusSim application development, it is imperative 
that any assets remain as flexible and extensible as possible or reusabihty may be 
diminished. Therefore, we chose to implement the unique aspects of BusSim 
applications using a component approach rather than a framework approach. This 
decision is further supported by the following observations. 

■ An application can only be based on one framework. Using the house 
analogy, if you Hke the first floor of one framework and the second floor of another, 
it is difficult or impossible to integrate the features of the two. Or, it is so costly as 
to erase the benefit of using a framework in the first place. Likewise with 
application frameworks. You can only use one framework when building an 
apphcation You can't mix and match features from multiple frameworks, so any 
framework that we developed would have to compete against existing and future 
frameworks. With components, however, you can mix and match from multiple 
vendors. 

■ Components are less platform and development tool dependent, leaving 
more options open for development teams. An apphance hke a dishwasher is not 
restricted for use in a particular type of house. Similarly, component technologies 
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exist that are independent of platform and development tool. For example ActiveX 
can be used in almost every development environment for Windows and Java Beans 
components can be used on a wide variety of platforms. 

■ Frameworks become obsolete more quickly. Rapid emergence and evolution 
of technology has introduced a wealth of new feature requirements into application 
development. Frameworks that do not include the most current features become 
obsolete quickly. Components typically address a more focused feature set and are 
not as impacted by technology advances outside their core functionality areas. 

Based on these observations, we believe a combmed framework/component 
approach is optimal for achieving reuse. Figure 6 illustrates a development 
architecture approach in accordance with a preferred embodiment. 

Delivery Framework for Business Simulation 

This diagram illustrates an ideal solution where components are combined with an 
Application Framework and an Application Architecture to achieve maximum reuse 
and minimum custom development effort. The Application Architecture is added to 
provide communication support between the application interface and the 
components, and between the components. This solution has the following features: 

■ The components (identified by the icons) encapsulate key BusSim functionahty. 

■ The Application Architecture provides the glue that allows application-to- 
component and component-to-component communication. 

■ The Application Framework provides structure and base functionality that can be 
customized for different interaction styles. 

■ Only the appUcation interface must be custom developed. 

The next section discusses each of these components in further detail. 
The Business Simulation Toolset 

We have clearly defined why a combined component/firamework approach is the 
best solution for delivering high-quality BusSim solutions at a lower cost. Given 
that there are a number of third party fi-ameworks aheady on the market that provide 
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delivery capability for a wide variety of platforms, the TEL project is focused on 
defining and developing a set of components that provide unique services for the 
development and delivery of BusSim solutions. These components along with a set 
of design and test workbenches are the tools used by instructional designers to 
5 support activities in the four phases of BusSim development. We call this suite of 
tools the Business Simulation Toolset. Following is a description of each of the 
components and workbenches of the toolset. 

Components 

10 A Component can be thought of as a black box that encapsulates the behavior and 
data necessary to support a related set of services. It exposes these services to the 
outside world through published interfaces. The pubUshed interface of a component 
;0 allows you to understand what it does through the services it offers, but not how it 

y does it. The complexity of its implementation is hidden from the user. The 

^: : 15 following are the key components of the BusSim Toolset. 

ill 

■ Domain Component - provides services for modeling the state of a simulation 
,P ■ Profiling Component - provides services for rule-based evaluating the state of a 

y simulation 

|1| ■ Transformation Component - provides services for manipulating the state of a 

IZ 20 simulation 

^"^^ ■ Remediation Component - provides services for the rule-based delivering of 

feedback to the student 

Domain Component 

25 The Domain Model component is the central component of the suite that facilitates 
communication of context data across the application and the other components. It 
is a modeling tool that can use industry-standard database such as Informix, Oracle, 
or Sybase to store its data. 

A domain model is a representation of the objects in a simulation. The objects are 
30 such pseudo tangible things as a lever the student can pull, a form or notepad the 
student fills out, a character the student interacts with in a simulated meeting, etc. 
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They can also be abstract objects such as the ROI for a particular investment, the 
number of times the student asked a particular question, etc. These objects are 
called entities. Some example entities include: 

■ Vehicles, operators and incidents in an insurance domain 

5 ■ Journal entries, cash flow statements and balance sheets in a financial accounting 
domain 

■ Consumers and purchases in a marketing domain 

An entity can also contain other entities. For example, a personal bank account 
10 entity might contain an entity that represents a savings account. Every entity has a 
set of properties where each property in some way describes the entity. The set of 
properties owned by an entity, in essence, define the entity. Some example 
properties include: 

■ An incident entity on an insurance application owns properties such as 
15 "Occurrence Date", "Incident Type Code", etc. 

■ A joumal entry owns properties such as "Credit Account", "Debit Account", and 
"Amount" 

■ A revolving credit accoimt entity on a mortgage application owns properties such 
as "Outstanding Balance", "Available Limit", etc. Figure 7 illustrates a small 

20 segment of a domain model for claims handlers in the auto insurance industry in 
accordance with a preferred embodiment. 

Example Domain Model 

The domain model is created by the instructional designer in a visual editing design 
25 tool called the Knowledge Workbench. The designer creates the objects of the 
domain model using generic entities and properties; that is, not having specific 
values associated with the entities and properties. 

At runtime, an application's domain model is instantiated so that every entity and 
30 property is given a particular value that makes it unique. The result of a domain 
model instantiation is called the domain. The values of a domain's entities and 
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properties can change throughout the course of the simulation based on student 
actions and updates from other components. Figure 8 illustrates an instantiated 
domain model in accordance with a preferred embodiment. 

Example Domain 

Creating a domain model in data rather than in code facilitates reuse of the 
components in multiple applications in multiple domains without code changes. For 
example, a typical apphcation in the Financial Services domain would have to define 
classes in code such as 'Customer', 'Account', etc. An Insurance Domain 
application might have classes such as 'Customer', 'Incident', 'Prior Policy', etc. 
To be able to perform analysis on any of these classes, the analysis logic must be 
coded to recognize the classes. This requires all logic to be custom-coded for every 
application; an effort-intensive undertaking that demands a high degree of technical 
skill. 

By modeling the domain in data using generic objects, we can build standard generic 
analysis capability that can be appUed to the domain. This allows implementation of 
analysis logic with much less effort by people with a low degree of technical skill. 
Functional experts can create the objects of the domain and apply various types of 
analysis from a pallet. All of this is accomplished in a visual development 
environment that supports the designer with visual feedback and only allows vaUd 
designs to be created. 

Profiling Component 

In the simplest terms, the purpose of the Profiling Component is to analyze the 
current state of a domain and identify specific things that are true about that domain. 
This information is then passed to the Remediation Component which provides 
feedback to the student. The Profiling Component analyzes the domain by asking 
questions about the domain's state, akin to an investigator asking questions about a 
case. The questions that the Profiler asks are called profiles. For example, suppose 
there is a task about building a campfire and the student has just thrown a match on a 
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pile of wood, but the fire didn't start. In order to give useful feedback to the student, 
a tutor would need to know things like: was the match lit?, was the wood wet?, was 
there kindling in the pile?, etc. These questions would be among the profiles that the 
Profiling Component would use to analyze the domain. The results of the analysis 
5 would then be passed off to the Remediation Component which would use this 
information to provide specific feedback to the student. 

Specifically, a profile is a set of criteria that is matched against the domain. The 
purpose of a profile is to check whether the criteria defined by the profile is met in 

10 the domain. Using a visual editing tool, instructional designers create profiles to 
identify those things that are important to know about the domain for a given task. 
During execution of a BusSim application at the point that feedback is requested 
either by the student or pro-actively by the appUcation, the set of profiles associated 
with the current task are evaluated to detemiine which ones are true. Example 

15 profiles include: 

■ Good productions strategy but wrong Break-Even Formula 

■ Good driving record and low claims history 

■ Correct Cash Flow Analysis but poor Return on Investment (ROI) 

20 A profile is composed of two types of structures: characteristics and collective 

characteristics. A characteristic is a conditional (the z/half of a rule) that identifies 
a subset of the domain that is important for determining what feedback to deliver to 
the student. Example characteristics include: 

■ Wrong debit account in transaction 1 
25 ■ Perfect cost classification 

■ At Least 1 DUX in the last 3 years 

■ More than $4000 in claims in the last 2 years 

■ More than two at- fault accidents in 5 years 

30 A characteristic's conditional uses one or more atomics as the operands to identify 
the subset of the domain that defines the characteristic. An atomic only makes 
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reference to a single property of a single entity in the domain; thus the term atomic. 
Example atomics include: 

■ The number of DUI's >= 1 

■ ROI>10% 

5 ■ Income between $75,000 and $ 1 10,000 

A collective characteristic is a conditional that uses multiple characteristics and/or 
other collective characteristics as its operands. Collective characteristics allow 
instructional designers to build richer expressions (i.e., ask more complex 
10 questions). Example collective characteristics include: 

■ Bad Household driving record 

■ Good Credit Rating 

■ Marginal Credit Rating 

■ Problems with Cash for Expense transactions 
15 ■ Problems with Sources and uses of cash 

Once created, designers are able to reuse these elements within multiple expressions, 
which significantly eases the burden of creating additional profiles. When building 
a profile from its elements, atomics can be used by multiple characteristics, 
20 characteristics can be used by multiple collective characteristics and profiles, and 
collective characteristics can be used by multiple collective characteristics and 
profiles. 

Figure 9 illustrates an insurance underwriting profile in accordance with a preferred 
embodiment. 

25 

Example Profile for Insurance Underwriting 

Transformation Component 

30 Whereas the ProlBling Component asks questions about the domain, the 

Transformation Component performs calculations on the domain and feeds the 
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results back into the domain for further analysis by the ProjfiUng Component. This 
facilitates the modeling of complex business systems that would otherwise be very 
difficult to implement as part of the application. Within the Analysis phase of the 
Interface/Analysis/Interpretation execution flow, the Transformation Component 
5 actually acts on the domain before the ProfiUng Component does its analysis. 
The Transformation Component acts as a shell that wraps one or more data 
modeling components for the purpose of integrating these components into a 
BusSim application. The Transformation Component facilitates the transfer of 
specific data from the domain to the data modeling component (inputs) for 
10 calculations to be performed on the data, as well as the transfer of the results of the 
calculations from the data modeling component back to the domain (outputs). 
Figure 10 illustrates a transformation component in accordance with a preferred 
embodiment. 

15 The data modeling components could be third party modeling environments such as 
spreadsheet-based modeling (e.g., Excel, Formulal) or discrete time-based 
simulation modehng (e.g., PowerSim, VenSim). The components could also be 
custom built in C-H-, VB, Access, or any tool that is ODBC compliant to provide 
unique modeling environments. Using the Transformation Component to wrap a 

20 third party spreadsheet component provides an easy way of integrating into an 
application spreadsheet-based data analysis, created by such tools as Excel. The 
Transformation Component provides a shell for the spreadsheet so that it can look 
into the domain, pull out values needed as inputs, performs its calculations, and post 
outputs back to the domain. 

25 For example, if the financial statements of a company are stored in the domain, the 
domain would hold the baseline data Uke how much cash the company has, what its 
assets and liabilities are, etc. The Transformation Component would be able to look 
at the data and calculate additional values like cash flow ratios, ROI or NPV of 
investments, or any other calculations to quantitatively analyze the financial health 

30 of the company. Depending on their complexity, these calculations could be 
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performed by pre-existing spreadsheets that a cUent has already spent considerable 
time developing. 

Remediation Component 

5 The Remediation Component is an expert system that faciUtates integration of 
inteUigent feedback into BusSim applications. It has the following features: 

■ Ability to compose high quality text feedback, 

■ Ability to compose multimedia feedback that includes video and/or audio. 

■ AbiUty to include reference material in feedback such as Authorware pages or 
10 Web Pages. 

■ Ability to actively manipulate the users deliverables to highhght or even fix 
users' errors. 

■ A proven remediation theory embedded in its feedback composition algorithm. 

■ Allows integration of digital assets into the Remediation of a training or IPS 
15 apphcation. 

The Remediation model consists of three primary objects: 

■ Concepts 

■ Coach Topics 
20 ■ Coach Items 

Concepts are objects that represent real- world concepts that the user will be faced 
with in the interface. Concepts can be broken into sub-concepts, creating a 
hierarchical tree of concepts. This tree can be arbitrarily deep and wide to support 
25 rich concept modeling. Concepts can also own an arbitrary number of Coach 
Topics. 

Coach Topics are objects that represent a discussion topic that may be appropriate 
for a concept. Coach Topics can own an arbitrary number of Coach Items. 
Coach Items are items of feedback that may include text, audio, video, URL's, or 
30 updates to the Domain Model, Coach Items are owned by Coach Topics and are 
assembled by the Remediation Component algorithm. 
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Details of the Remediation Component algorithm for feedback is discussed in The 
Intelligent Coaching Agent Tool whitepaper and can be found on the Knowledge 
Exchange at the Technology Enabled Learning ETA Home Page. 

5 Workbenches 

The BusSim Toolset also includes a set of workbenches that are used by 
instructional designers to design and build BusSim applications. A workbench is a 
tool that facihtates visual editing or testing of the data that the BusSim Components 
use for determining an application's run-time behavior. The BusSim Toolset 
10 includes the following workbenches: 
Knowledge Workbench 

The Knowledge Workbench is a tool for the creation of domain, analysis and 
feedback data that is used by the BusSim Components. It has the following features: 

■ Allows the designer to 'paint' knowledge in a drag-and-drop interface. 

15 ■ Knowledge is represented visually for easy communication among designers. 

■ The interface is intelligent, allowing designers to only paint valid interactions. 

■ Designer's Task creations are stored in a central repository. 

■ The workbench supports check-in / check-out for exclusive editing of a task. 

■ Supports LAN-based or untethered editing. 

20 ■ Automatically generates documentation of the designs. 

■ Generates the data files that drive the behavior of the components. 

Simulated Student Test Workbench 

The Simulated Student Test Workbench is a tool for the creation of data that 
25 simulates student's actions for testing BusSim Component behaviors. It has the 
following features: 

■ The Test Bench generates a simulated application interface based on the Domain 
Model. 

■ The designer manipulates the objects in the Domain Model to simulate student 
30 activity. 



-47- 



■ The designer can invoke the components to experience the interactions the 
student will experience in production. 

■ The designer can fully test the interaction behavior prior to development of the 
appUcation interface. 

5 

Regression Test Workbench 

The Regression Test Workbench is a tool for replaying and testing of student 
sessions to aid debugging. It has the following features: 

■ Each student submission can be individually replayed through the components. 
10 "An arbitrary number of student submissions from the same session can be 

replayed in succession. 

■ Entire student sessions can be replayed in batch instantly. 

■ The interaction results of the student are juxtaposed with the results of the 
regression test for comparison. 

15 

Development Cycle Activities 
Design Phase 

The design phase of a BusSim appUcation is streamlined by the use of the 
Knowledge Workbench. The Knowledge Workbench is a visual editor for 
20 configuring the objects of the component engines to control their runtime behavior. 
The components are based on proven algorithms that capture and implement best 
practices and provide a conceptual framework and methodology for instructional 
design. 

25 In conceptual design, the workbench allows the designer to paint a model of the 

hierarchy of Concepts that the student will need to master in the activity. This helps 
the designer organize the content in a logical way. The visual representation of the 
Concepts helps to communicate ideas to other designers for review. The consistent 
look and feel of the workbench also contributes to a streamlined QuaHty Assurance 

30 process. In addition, standard documentation can be automatically generated for the 
entire design. 
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As the design phase progresses, the designer adds more detail to the design of the 
Concept hierarchy by painting in Coach Topics that the student may need feedback 
on. The designer can associate multiple feedback topics with each Concept. The 
designer also characterizes each topic as being Praise, Polish, Focus, Redirect or one 
of several other types of feedback that are consistent with a proven remediation 
methodology. The designer can then fill each topic with text, video war stories, 
Web page Hnks, Authorware links, or any other media object that can be delivered to 
the student as part of the feedback topic. 

As the designer's thoughts for the interface become clearer, she can begin to model 
the domain objects in the Knowledge Workbench. The student's world is 
constructed using objects in the Domain Model. 

The designer again uses the Knowledge Workbench to configure objects in the 
Transformation Component. The Transformation Component is used to perform 
calculations or other analysis of the student's domain. Lastly, the designer uses the 
workbench to configure objects in the Profihng Component. The ProfiKng 
Component examines the student's domain, looking for conditions that indicate what 
feedback topics are appropriate for delivery. 

More importantly, the Student Simulator Test Workbench allows the designer to 
exercise the designs. It allows the designer to manipulate the domain as if she were 
a student. The designer can interact with the simulated interface and invoke the 
component engines to see the feedback that the student would receive. This 
capability can also be utilized in a usability test such as a Conference Room Pilot. 
As the test student interacts with screen mock-ups, a facilitator can mimic his actions 
in the interface simulator and tell the student what the actual feedback will be. This 
results in much more rigorous testing prior to application construction. A big payoff 
is realized downstream in the form of reduced redesign after usability and cognition 
testing. 
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Throughout all these steps in the initial design, the workbench supports the design 
process by allowing the designer great flexibility within the framework of a proven 
methodology. This allows experienced users to design rich, reahstic interactions, 
and inexperienced users to become competent in a shorter time by leaming from the 
best practices embedded in the workbench. This greatly diminishes the 'blue sky' 
design problem. Also, since the designs can be tested prior to application 
construction, there is reduced rework after testing. Lastly, the visual knowledge 
representation enhances communication within the design team and greatly 
streamlines the QA process. 

Build Phase 

It is very clear how the tools support the Build Phase. The designs that the designer 
painted in the Knowledge Workbench drive the components at runtime. The 
application developer no longer has to write code that analyzes the student's work 
and provides feedback. The developer only has to build the interface and logic to 
report any student actions to the domain model. The components do the rest. What 
used to be the most difficult part of the build phase has been eliminated! 

There is no chance for a developer to misinterpret the feedback designs because she 
never has to interpret them. In fact, the developer doesn't even have to know 
anything about the feedback behavior as long as she is famihar with the domain 
model. This also means the skill level required to develop the application can be 
greatly reduced. It's not hard to teach someone how to paint a screen in Visual 
Basic or Delphi and call API fimctions to notify the Domain Model of student 
actions. 

In addition to the economies gained by the components, it is possible to use 
templates to further streamhne design and development of commonly used 
interactions. We have created templates for several common interactions. For 
example, Joumahzing of Transactions is an interaction that has appeared in several 
applications. We have built apphcation and Knowledge Workbench templates for 
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Joumalization. All one must do to create a new Journalize task is to add graphics for 
new Transactions and fill in new data into placeholders in the Knowledge 
Workbench, 

5 Test Phase 

The toolset greatly reduces effort during functionality testing. The key driver of the 
effort reduction is that the components can automatically track the actions of the 
tester without the need to add code support in the application. Whenever the tester 
takes an action in the interface, it is reported to the domain model. From there it can 
10 be tracked in a database. Testers no longer need to write down their actions for use 
in debugging; they are automatically written to disk. There is also a feature for 
attaching comments to a tester's actions. When unexpected behavior is encountered, 
the tester can hit a control key sequence that pops up a dialog to record a description 
of the errant behavior. 

15 

Of far greater impact is the abihty to replay the tester's actions automatically 
through the Regression Test Workbench. The designer does not need to spend hours 
trying to duplicate the error. She simply loads the tester's session into the 
Regression Test Workbench and replays it. In seconds the error is replicated and can 
20 be located and fixed using a variety of debugging utilities. After changes have been 
made, one more pass through the Regression Test Workbench verifies the fix. 

The major difficulties of usability and cognition testing are also addressed by the 
toolset. First, since student tracking is no longer a manual activity, the precision of 
25 fimctional testing can also be applied to usabiUty and cognition testing. Second, 
because of the increased rigor in the Conference Room Pilot, the risk of significant 
rework is greatly reduced. 



30 



Execution Phase 

During the Execution Phase, the components are deployed to the student's platform. 
They provide simulated team member and feedback functionality with sub-second 
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response time and error-free operation. If the client desires it, student tracking 
mechanisms can be deployed at runtime for evaluation and administration of 
students. This also enables the isolation of any defects that may have made it to 
production. 

5 

Scenarios for Using the Business Simulation Toolset 

A good way to gain a better appreciation for how the BusSim Toolset can vastly 
improve the BusSim development effort is to walk through scenarios of how the 
tools would be used throughout the development Hfecycle of a particular task in a 

10 BusSim appUcation. For this purpose, we'll assume that the goal of the student in a 
specific task is to joumahze invoice transactions, and that this task is within the 
broader context of learning the fundamentals of financial accounting. A cursory 
description of the task from the student's perspective will help set the context for the 
scenarios. Following the description are five scenarios which describe various 

15 activities in the development of this task. The figure below shows a screen shot of 
the task interface. 

Figure 11 illustrates the use of a toolbar to navigate and access appUcation level 
features in accordance with a preferred embodiment. A student uses a toolbar to 

20 navigate and also to access some of the appUcation-level features of the application. 
The toolbar is the inverted L-shaped object across the top and left of the interface. 
The top section of the toolbar allows the user to navigate to tasks within the current 
activity. The left section of the toolbar allows the student to access other features of 
the application, including feedback. The student can have his deUverables analyzed 

25 and receive feedback by clicking on the Team button. 

In this task, the student must journalize twenty-two invoices and other source 
documents to record the flow of budget dollars between internal accounts, (Note: 
"JoumaUzing", or "Journalization", is the process of recording journal entries in a 
general ledger from invoices or other source documents during an accounting period. 

30 The process entails creating debit and balancing credit entries for each document. 



-52- 



At the completion of this process, the general ledger records are used to create a trial 
balance and subsequent financial reports.) 

In accordance with a preferred embodiment, an Intelligent Coaching Agent Tool 
5 (ICAT) was developed to standardize and simplify the creation and delivery of 

feedback in a highly complex and open-ended environment. Feedback from a coach 
or tutor is instrumental in guiding the leamer through an apphcation. Moreover, by 
diagnosing trouble areas and recommending specific actions based on predicted 
student understanding of the domain student comprehension of key concepts is 

10 increased. By writing rules and feedback that correspond to a proven feedback 
strategy, consistent feedback is deUvered throughout the application, regardless of 
the interaction type or of the specific designer/developer creating the feedback. The 
ICAT is packaged with a user-fiiendly workbench, so that it may be reused to 
increase productivity on projects requiring a similar rule-based data engine and 

15 repository. 

Definition of ICAT In Accordance with a Preferred Embodiment 

The InteUigent Coaching Agent Tool (ICAT) is a suite of tools~a database and a 
Dynamic Link Library (DLL) run-time engine — used by designers to create and 
20 execute just-in-time feedback of Goal Based training. Designers write feedback and 
rules in the development tools. Once the feedback is set, the run-time engine 
monitors user actions, fires rules and composes feedback which describes the 
business deliverable. 

25 1. The ICAT Remediation Model 

The remediation model used within ICAT dynamically composes the most 
appropriate feedback to deHver to a student based on student's previous responses. 
The ICAT model is based on a theory of feedback which has been proven effective 
by pilot results and informal interviews. The model is embodied in the object model 
30 and algorithms of the ICAT. Because the model is built into the tools, all feedback 



-53- 



created with the tool will conform to the model 

II. The Role of ICAT in Student Training 

The ICAT plays two roles in student training. First, the ICAT is a teaching system, 
5 helping students to fully comprehend and apply information. Second, ICAT is a 
gatekeeper, ensuring that each student has mastered the material before moving on to 
additional information. 

III. The Functional Definition of the ICAT 

10 The ICAT is a self contained module, separate from the application. Separating the 
ICAT from the apphcation allows other projects to use the ICAT and allows 
designers to test feedback before the apphcation is complete. The ICAT Module is 
built on six processes which allow a student to interact effectively with the interface 
to compose and deliver the appropriate feedback for a student's mistakes. 

15 

IV. The ICAT Development Methodology for Creating Feedback 

The ICAT development methodology is a seven step methodology for creating 
feedback. The methodology contains specific steps, general guidelines and lessons 
learned from the field. Using the methodology increases the effectiveness of the 
20 feedback to meet the educational requirements of the course. 

V. Components 

The processes each contain a knowledge model and some contain algorithms. Each 
process has specific knowledge architected into its design to enhance remediation 
25 and teaching. 

VI. Testing Utilities, Reports and Methodology 

There is a suite of testing tools for the ICAT. These tools allow designers and 
developers test all of their feedback and rules. In addition, the utilities let designers 
30 capture real time activities of students as they go through the course. 
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Expert Remediation Model Within the Tools 

The tools and run-time engine in accordance with a preferred embodiment include 
expert knowledge of remediation. These objects include logic that analyzes a 
5 student's work to identify problem areas and deliver focused feedback. The 
designers need only instantiate the objects to put the tools to work. Embodying 
expert knowledge in the tools and engine ensures that each section of a course has 
the same effective feedback structure in place. 

10 Any project which is creating a Goal-Based Scenario (GBS) business simulation or 
an Integrated Performance Support (IPS) system to help users understand and create 
business deliverables can profit from technology in accordance with a preferred 
embodiment. A GBS allows students to learn in a comprehensive simulated 
environment. Students work in a simulated environment to accomplish real world 

15 tasks, and when they make mistakes, remediation is provided to help identify and 
correct the mistakes. The hands-on experience of the simulated environment and 
the timely remediation account for the high retention rate from subjects presented 
utilizing a system in accordance with a preferred embodiment. A system in 
accordance with a preferred embodiment can be used in conjunction with an IPS to 

20 help users develop deliverables. If a customer service representative (CSR) is 

completing a form while conducting a phone conversation, the ICAT can be used to 
observe how the task is completed to provide a live analysis of mistakes. 



A file structure in accordance with a preferred embodiment provides a standard 
25 system environment for all applications in accordance with a preferred embodiment. 
A development directory holds a plurality of sub-directories. The content in the 
documentation directory is part of a separate installation from the architecture. This 
is due to the size of the documentation directory. It does not require any support 
files, thus it may be placed on a LAN or on individual computers. 



30 
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When the architecture is installed in accordance with a preferred embodiment, the 
development directory has an _Arch, _Tools, _UtiUties, Documentation, QED, and 
XDefault development directory. Each folder has its own directory structure that is 
inter-linked with the other directories. This structure must be maintained to assure 
5 consistency and compatibility between projects to clarify project differences, and 
architecture updates. 

The _Arch directory stores many of the most common parts of the system 
architecture. These files generally do not change and can be reused in any area of 
the project. If there is common visual basic code for applications that will 
continuously be used in other applications, the files will be housed in a folder in this 
directory. 

The sub-directories in the _Arch directory are broken into certain objects of the main 
project. Object in this case refers to parts of a project that are commonly referred to 
within the project. For example, modules and classes are defined here, and the 
directory is analogous to a library of fiinctions, APIs, etc. . . that do not change. For 
example the IcaObj directory stores code for the InteUigent Coaching Agent (ICA), 
The InBoxObj directory stores code for the InBox part of the project and so on. The 
file structure uses some primary object references as file directories. For example, 
the IcaObj directory is a component that contains primary objects for the ICA such 
as fimctional forms, modules and classes. 

The BrowserObj directory contains modules, classes and forms related to the 
25 browser fixnctionahty in the architecture. 

The HTMLGlossary directory contains code that is used for the HTML reference 
and glossary component of the architecture. 
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The IcaObj directory contains ICA functional code to be used in an application. 
This code is instantiated and enhanced in accordance with a preferred embodiment. 

The InBoxObj directory contains code pertaining to the inbox functionality used 
within the architecture. Specifically, there are two major components in this 
architecture directory. There is a new .ocx control that was created to provide 
functionality for an inbox in the apphcation. There is also code that provides 
support for a legacy inbox apphcation. The PracticeObj directory contains code for 
the topics component of the architecture. The topics component can be implemented 
with the HTMLGlossary component as well. 

The QmediaObj directory contains the components that are media related. An 
example is the QVIDctrl.cls, The QVIDctrl is the code that creates the links 
between QVID files in an application and the system in accordance with a preferred 
embodiment. 

The SimObj directory contains the Simulation Engine, a component of the 
application that notifies the tutor of inputs and outputs using a spreadsheet to 
facilitate communication. 

The StaticObj directory holds any component that the apphcation will use statically 
from the rest of the application. For example, the login form is kept in this folder 
and is used as a static object in accordance with a preferred embodiment. 

The SysDynObj directory contains the code that allows the Systems Dynamics 
Engine (Powersim) to pass values to the Simulation Enghie and return the values to 
the tutor. 
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The VBObj directory contains common Visual Basic objects used in applications. 
For example the Now What, Visual Basic Reference forms, and specific message box 
components are stored in this folder. 

5 The _Tools directory contains two main directories. They represent the two most 
used tools in accordance with a preferred embodiment. The two directories provide 
the code for the tools themselves. The reason for providing the code for these tools 
is to allow a developer to enhance certain parts of the tools to extend their ability. 
This is important for the current project development and also for the growth of the 
10 tools. 

The Icautils directory contains a data, database, default, graphics, icadoc, and 
testdata directory. The purpose of all of these directories is to provide a secondary 
working directory for a developer to keep their testing environment of enhanced 
15 Icautils applications separate from the project application. It is built as a testbed for 
the tool only. No application specific work should be done here. The purpose of 
each of these directories will be explained in more depth in the project directory 
section. The TestData folder is unique to the _Tools/ICAUtils directory. It contains 
test data for the regression bench among others components in ICAUtils, 

20 

Utilities 

The Utilities directory holds the available utilities that a Business Simulation project 
requires for optimal results. This is a repository for code and executable utilities that 
developers and designers may utilize and enhance in accordance with a preferred 
25 embodiment. Most of the utilities are small applications or tools that can be used in 
the production of simulations which comprise an executable and code to go with it 
for any enhancements or changes to the utility. If new utilities are created on a 
project or existing utilities are enhanced, it is important to notify the managers or 
developers in charge of keeping track of the Business Simulation assets. Any 
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enhancements, changes or additions to the Business Simulation technology assets 
are important for future and existing projects. 

Documentation 

A Documentation directory is used to store pertinent documentation. The 
5 documentation directory is structured as follows. Most of the directories are labeled 
after the specific information held within them. The following is a list of all the 
documentation directories and a description of what is contain in each, 

Ref Website - This directory contains The Business Simulation Reference website, 
1 0 which is a general reference for many things. If the website has not been set up for 
users on a LAN or website, all you need to do is go into the root directory of website 
and double click on index.htm. This is the main page for the site. 

Components - This directory contains any documentation on classes and modules 
15 that are used in the archtecture. For example there are docimients here on the 
ICAMeeting class, the hibox class etc. 

Database - This directory contains any documents describing the databases that are 
included and used in the Architecture. For example the ICAObj overview doc 
20 contains a description of the model and each element in the database. 

HTML Component - This directory contains relevant documentation about the 
HTML part of the architecture. 

25 Process Models - This directory should contain the documents that describe the 
process of the application or related information. 

ReferenceApp - This directory contains documents with descriptions and views of 
the reference app. (QED) for explanation and documentation. Testing conditions are 
30 stored in the Testing directory. 
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Standards&TempIates - This directory contains any type of architecture relevant 
coding standard documents or templates that a developer is required to follow. 

5 UserGuides- This directory has 6 sub-directories. Each one of these sub-directories 
contains user guides for a given tool or component in accordance with a preferred 
embodiment which include user guides for the architecture, the Tutor Suite, ICA 
Utilities, the simulation Engine and the System Dynamics Engine, There is also a 
directory for other documentation that contains user guides for any other tools or 
1 0 code like third party controls etc, 

WorkFlows - This directory contains the WF_Develop,doc which includes the 
workflow documentation for an application. 

Project Directory 

1 5 The sample project directory, QED has the same structure that a real project would 
be designed after. The QED directory has all custom architecture code, databases, 
spreadsheets, and any other application specific files stored in it. The QED project 
directory stores a Design and SrcVB directory. The Design directory contains all 
relevant files for a designer. The SrcVB directory is used for a developer. 

20 

The root directory of the Design and SrcVB directory contain a few important files 
to note. Both have two .rtf files, a few log files and an .ini file. The .rtf files are the 
feedback that is output from the tutor, the logs are also output from the tutor and the 
.ini file is for ICAUtils initialization. The design directory has three subdirectories 
25 that contain a data directory, which stores .xls files, sim models, and any other 

important data like html and video. It also has a database directory that holds any 
relevant databases for development and application use. The last directory is the 
icadoc directory which includes all .tut files or .ica files, which are both created with 
the tutor. 

30 
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The SrcVB directory stores all of the directories previously described. The reason 
for duplicating the data and database directories is to assure that a developer does 
not interfere with the designer's files. The developer tends to not do as much design 
work and can easily corrupt files. This dupUcation of directories provides a safer 
5 environment for the developer to test in. As was mentioned above, the developer 
tends to have a lot more to do with the application build than the design so there 
needs to be more content in the SrcVB directory. The SrcVB directory also contains 
an ,exe and .vbp file which are created in a developers visual basic application. 

10 The following are directories found in the SrcVB directory that are not found in the 
Design directory followed by a short definition: 

The _CustomArch directory contains any appUcation specific architecture. Look in 
the QED folder for an example. 

15 

The _CustomDistribution directory contains any files that need to be distributed 
with the application. 

The Default directory contains any backup files that might need to be copied and 
20 reused later. Some files occasionally are corrupted and need to be replaced. 

The Fonts directory contains appUcation specific font libraries. 

The Graphics directory contains any relevant graphics for the application. 

25 

The Help directory contains all files for a help reference layer in the appHcation. 
This can be implemented in many ways but most commonly in an HTML form. 

The Saved directory is for saved information that is produced by the appUcation, 
30 This can be used for saving student information or saving temporary information for 
the application steps. 
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The StudentData directory is for storing any relevant student data, lists of students, 
their personal information or any relevant student data that needs to be saved. 

5 XDefault Development: 

The XDefault Development environment is used to provide a shell for any new 
project. A developer would rename this directory as an acronym of the project. 
QED is the default for the installation sample application. The XDefault 
development directory is a shell and serves as a building block for a startup project. 
10 A good idea is to use the QED sample appHcation and build the XDefault 
Development project with the sample code in QED. 

Shared Development 

The last directory to be mentioned is the shared development directory which is 
1 5 placed on a LAN or central network area and is shared by all designers and 

developers of a project to assure that files in the project are up to date, managed 
properly and appropriately synchronized. There are many databases and files that 
will be shared in accordance with a preferred embodiment. These files need to be 
shared and have a location that someone can edit without having to worry about 
20 merging files later. A source control program is used to restrict access to a file to 
one appUcation at a time. 

The ICAT Model of Remediation 

The ICAT has a model of remediation architected into the system in accordance 
with a preferred embodiment. Feedback should help students complete tasks and 
25 learn the underlying concepts. To achieve this goal, the ICAT reviews student's 
work with the following objectives in mind. 



30 



Identify student misconceptions 

Identifying that a student does not understand a topic and then clearly explaining it 
is the goal of human and computer tutors alike. Human tutors, however, have many 
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more clues-including facial expressions and body language—to help them identify 
student misconceptions. The computer tutor is much more limited and can only 
view the outputs— such as documents and reports— the student produces. If a 
computer tutor is looking for a misunderstanding about debits and credits, the 
5 computer analyzes all the mistakes a student made concerning debits and credits and 
tries to identify what misunderstanding would account for this pattem of mistakes. 

Identify what students should fix 

If the coach cannot diagnose a student's misconception, or cannot do it with 100% 
1 0 accuracy, the coach must at least tell the student what he did wrong so that he can 
correct it. If at all possible, the coach should identify groups or types of problems 
13 the student is making so that the student can generalize the solution and answer. 

5^ Prompt students to reflect on mistakes 

lU 1 5 When identifying problems, the tutor needs to prompt the student to reflect on a 
In problem and start to point the student towards the answer. The tutor should not tell 

JL, the student the answer, but instead should attempt to provide an appropriate answer 

ffl or give the student a question to think about. 

W 20 Reinforce correct concepts and ideas 

Once a student has gotten the correct answer, it is important to reinforce the 
learning. Students may feel uncertain about their understanding even after he has 
gotten the answer correct. To reinforce the student's understanding of the concept 
and provide a confidence boost, the tutor should walk the student through the 

25 answer so that it is completely understood. These goals are not just the goals of a 
computer tutor, but they are the goals of a human tutor as well. All tutors must look 
at a student's work to help identify and correct errors as well as learn the material. 
One of the most difficult tasks facing a tutor is the difficult task of balancing the 
appropriate amount of assistance provided the student to complete the task with the 

30 requirement to help the student leam the material. 
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Model of Feedback 

A preferred embodiment utilizes feedback to address the balancing task. The theory 
is centered on the idea of severity. Severe errors require severe feedback while mild 
errors require mild feedback. If a student writes a paper on the wrong subject, a 
5 human tutor will spend little time reviewing the paper, but instead, identify it as a 
serious mistake and ask the student to rewrite the paper. If the student simply 
misses one paragraph of the argument, then the tutor will focus the student on that 
paragraph. Finally, if the paper is correct except for a couple of spelling mistakes, 
the tutor will point out the specific mistakes and ask the student to correct them. The 
10 point is that because a tutor and a student do not want to waste each others' time, 
they will match the severity of the error with the severity of the feedback. 

In the ICAT model of feedback, there are four levels of severity of error and four 
corresponding levels of feedback. The tutor goes through the student's work, 
1 5 identifies the severity of the error and then provides the corresponding level of 
feedback. 



Educational Categories of Feedback 


ERROR 


FEEDBACK 


Error Type 


Description 


Feedback 
Type 


Description 


1. None 


No errors exist. 
The student's 
work is perfect. 


1, Praise 


Confirmation that the student 
completed the task correctly. 

Example: 

Great. You have journalized 
all accounts correctly. I am 
happy to see you recognized 
we are paying for most of our 
bills "on account". 
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2. 


Syntactic 


There may be 


2. 


PoHsh 


Tells the student the specific 






spelling 






actions he did incorrectly, 






mistakes or 






and oossiblv correct them for 






other syntactic 






him. 






errors. As a 












designer, you 






Example: 






should be 






There are one or two errors 






confident that 






in your work. It looks like 






the student will 






you misclassified the 






have mastered 






purchase of the fax as a cash 






the material at 






purchase when it is really a 






this point. 






purchase on account. 


3. 


Local 


A naraeranh of 


3. 


Focus 








a paper is 






of his work. Point out that 






missing or the 






he does not understand at 






student ha*? 






lP3^lt OTIP tnaiOT* CC\T\CPT\f 






made a number 












of mistakes all 






Example; 






in one area. 






Looking over your work, I 






The student 






see that you do not 






clearly does not 






understand the concent of 






understand this 






"on account" Whv don't 






area. 






you review that concept and 












review your work for errors. 


4. 


Global 




4 










written on the 






activitv and tell the student to 






wrong subject 






review main concepts and 






or there are 






retry the activity. 






mistakes all 












over the 






Example: 






student's work 






There are lots of mistakes 

1 
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which indicates 




throughout your work. You 




he does not 




need to think about what type 




understand most 




of transaction each source 




of the concepts 




document represents before 




in the activity. 




joumaUzing it. 



Returning to the analogy of helping someone write a paper, if the student writes on 
the wrong subject, this as a global error requiring redirect feedback. If the student 
returns with the paper rewritten, but with many errors in one area of the paper, focus 
5 feedback is needed. With all of those errors fixed and only spelling mistakes- 
syntactic mistakes— polish feedback is needed. When all syntactic mistakes were 
corrected, the tutor would retum praise and restate why the student had written the 
correct paper. 

1 0 Focusing on the educational components of completing a task is not enough. As 
any teacher knows, student will often try and cheat their way through a task. 
Students may do no work and hope the teacher does not notice or the student may 
only do minor changes in hope of a hint or part of the answer. To accommodate 
these administrative functions, there are three additional administrative categories of 

15 feedback. 



Administrative Categories of Feedback 


Error 


Description 


Feedback 


Description 


No work 


The student has made 


Mastermind 


Tell the student that he has 


done since 


no changes since the 




done no work and that a 


last review 


last time he asked for 




substantial amount of work 




the tutor to review his 




needs to be completed before 




work. 




review. 








Example: 








You have done no work since I 
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last reviewed your work. 








Please try and correct at least 








three joumal entries before 








asking me to review your work 








again. 


All work is 


If a designer wants to 


Incomplete- 


State that the student has not 


not 


give an interim report 


continue 


completed all of the work 


complete 


of how the student is 




required, but you will review 


but a 


doing before everything 




what the student has done so 


substantial 


is done, they he would 




far. 


amount of 


use incomplete— 






work has 


continue. 




Example: 


been done 






It looks like you have not 
finished journalizing, but I will 
review what you have done up 
to this point. The first three 
entries are correct. 


All work is 


If a user has not 


Incomplete- 


State that nothing has been 


not 


completed enough 


stop 


attempted and point to the first 


complete 


work to receive 




action to be taken. 


and a 


feedback, this category 






substantial 


is used. 




Example: 


amount of 






It looks like you have done no 


work is not 






work joumalizing. Why don't 


complete 






you start by trying to 
journalize the fax purchase. 



The administrative and the educational categories of feedback account for 
every piece of feedback a designer can write and a student can receive. To 
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provide a better understanding of how the feedback works together, an 
example is provided below. 

Feedback Example 

5 The following example is a GBS training application in which new finance 

professionals are taught the fundamentals of finance management. A student has a 
toolbar to navigate and also to access some of the appUcation-level features of the 
application. The toolbar is the L-shaped object across the top and left of the 
interface. The top section of the toolbar allows the user to navigate to tasks within 
10 the current Activity. The left section of the toolbar allows the student to access 
other features of the appUcation, including feedback. The student can have his 
deliverables analyzed and receive feedback by cUcking on a team button. 

In this task, the student must joumahze twenty-two invoices and other source 
15 documents to record the flow of budget dollars between internal accounts. (Note: 
"Joumalizing", or "JoumaUzation", is the process of recording joumal entries in a 
general ledger fi"om invoices or other source documents during an accounting 
period. The process entails creating debit and balancing credit entries for each 
document. At the completion of this process, the general ledger records are used to 
20 create a trial balance and subsequent financial reports.) The student has several 

controls on the screen that must be manipulated to complete the task. The upper left 
area of the screen shows the current transaction. Each transaction has a 
corresponding joumal entry. The bottom of the screen shows the current joumal 
entry. The Top two lines of the joumal entry are for Debits (DR) and the bottom 
25 two Unes are for Credits (CR). As the student uses the *Back' and 'Next' buttons to 
page through the transactions, the joumal entry is also paged to stay in sync. 

Figure 12 is a GBS display in accordance with a preferred embodiment. The upper 
right area of the screen shows the account list. There are four types of accounts: 
30 Assets, Liabilities & Equity, Revenues, and Expenses. The user cUcks on one of the 
tabs to show the accounts of the corresponding type. The student joumalizes a 
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10 



transaction by dragging an account from the account list onto the journal entry 
Debits or Credits, The student then enters the dollar amounts to debit or credit each 
account in the entry. In the interface, as in real life, the student can have multi- 
legged journal entries (i.e., debiting or crediting multiple accounts). 

A Toolbar 1200 and the first transaction of this Task 1210 appear prominently on the 
display. The student can move forward and back through the stack of transactions. 
For each transaction, the student must identify which accounts to debit and which to 
credit. When the student is done, he clicks the Team button. 



Figure 13 is a feedback display in accordance with a preferred embodiment. The 
student may attempt to outsmart the system by submitting without doing anything. 
The ICAT system identifies that the student has not done a substantial amount of 
work and returns the administrative feedback depicted in Figure 13. The feedback 
1 5 points out that nothing has been done, but it also states that if the student does some 
work, the tutor will focus on the first few journal entries. 

Figure 14 illustrates a display in which a student has made some mistakes in 
accordance with a preferred embodiment. The student tries to joumalize the 

20 transaction depicted in Figure 14 which reflects the capital needed to start the 

business. The student attempts to joumalize the transaction by debiting the paid-in 
capital account and crediting the cash account for $210,000. Similarly, the student 
attempts to joumalize the pxirchase of Government Bonds by debiting accoimts 
receivable and crediting cash for $150,000 as shown in Figure 15. Figure 15 

25 illustrates a journal entry simulation in accordance with a preferred embodiment. 

Figure 16 illustrates a simulated Bell Phone Bill journal entry in accordance with a 
preferred embodiment. The journal entry is accompUshed by debiting UtiUties 
Expenses and Crediting Cash for $700 each. 

30 
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Figure 17 illustrates a feedback display in accordance with a preferred embodiment. 
After attempting to journalize the first three transactions, the student submits his 
work and receives the feedback depicted in Figure 17. The feedback starts by 
focusing the student on the area of work being evaluated. The ICAT states that it is 
5 only looking at the first three journal entries. The feedback states that the first two 
entries are completely wrong, but the third is close. If the student had made large 
mistakes on each of the first three transactions, then the ICAT may have given 
redirect feedback, thinking a global error occurred. The third bullet point also 
highlights how specific the feedback can become, identifying near misses. 

10 

Figures 18 and 19 illustrate a feedback display in accordance with a preferred 
embodiment. 

As a student attempts to correct transactions one and two unsuccessfiiUy, the tutor 
starts to provide hints, stating that the student should debit an asset account and 
15 credit an equity account. The ICAT continues to focus on the errors in the first three 
source documents and is giving progressively more specific hints. 

Figure 20 illustrates a feedback display in accordance with a preferred embodiment. 
With the specific hints provided as illustrated in Figure 19, the student correctly 

20 journalizes the source document. The ICAT, however, continues to focus the 

student on these first three journal entries as illustrated in Figure 20. The student 
finally completes the first three entries correctly. The feedback illustrated in Figure 
20 informs the student of his success and instructs him to try to complete the rest of 
the transaction before submitting his dehverable again. This example illustrates the 

25 use of an effective technique called "baby-stepping". The student is helped through 
a small portion of the work to get him introduced to the concepts and the interface. 
After completing this, he is forced to attempt all of the remaining work before 
getting substantive feedback. This technique can be used to mimic the kind of 
interactions one could expect to receive from a mentor in real life. The three 

30 transactions above show a tiny fraction of the depth of student analysis and richness 
of remediation that the ICAT is capable of delivering. 
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As mentioned earlier in the Remediation Model section, the tutor plays two roles in 
any course. First, the tutor reviews the student's work and helps him/her understand 
the task and the associate concepts. Second the tutor is gatekeeper between 
5 sections. The tutor will not allow students to proceed to the next section of the 
course until they have gotten the current section correct. To monitor student 
progress, the course has been broken into two components: 

Activity 

An activity is a business event, such as planning a company's financials or closing 
10 the books. Business events set the context of the course. Students leam the content 
so that they can complete the goals associates with each business event. The power 
of a GBS is in how it embeds the content a student needs to leam within the context 
of the business events. 

Task 

15 A task is a business deliverable that must be completed as part of a business event. 
Example tasks include completing journal entries while closing the books. There 
may be many Tasks in an activity, just as there may be many deliverables required 
to react to a business event in the real world. Deliverables produced in this 
application include a break-even analysis, a transaction journal, a cost report, and a 

20 ratio analysis. The role of the tutor is to help the students complete the business 
deliverables associated with any business event. Students can always go backward, 
but they are limited from going forward, until the ICAT says that the business 
deliverable meets the required specifications. It is useful to think of the ICAT as a 
boss who reviews your work. The boss will not let you go on to the next task, or 

25 business dehverable, until you have correctly completed the current task. To help 
explain the concepts of an activity and task, here is a description of an ICAT 
implementation in accordance with a preferred embodiment. 

A training application utilizing ICAT for a large product company is presented as an 
30 example. The training application is a revision of the first semester of a two year 
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fmancial training program. Students learn finance by managing a simulated bicycle 
company for three years and using finance to solve business problems. At four 
places in the course, the students come together to present their analyses of the 
business. These presentations are Kve presentations to real business executives. 

5 

In preparation for the pitches, the students complete computer-based modules. 
There are two major sections to each module, the accounting concepts and the 
activities. Students leam the concepts and ideas in the accounting concepts and 
apply the concepts in the activities. All of the modules together represent the 
10 different phases associated with running a business: Start Operations, Analyze 
Operations and Improve Operations, Each computer-based activity represents a 
business event, such as closing the books of the company. These business events 
m provide context for the content the students leam in the course. In this way, 

1^ students not only leam what the concepts are but when, how and why they should 

15 use them. 



Business Events — ^Activities 


1 , Financial Planning 


4. Closing the Books 


2. Recording Transactions 


5, Analyze the Books 


3 . Recording Transactions 


6. Improve Operations 



Figure 21 illustrates a simulation display in accordance with a preferred 

20 embodiment. 

To show how the business events impact the company on a day to day basis, 
students complete a set of deliverables associated with each business event. The 
business deliverables students create in the training application are varied in form 
and content. Some example business deliverables are listed below in accordance 

25 with a preferred embodiment. 
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1 . An analysis of proforma financial statements 

Students perform break-even analysis to determine which of twelve business 
strategies to pursue. 

2. Journal entries 

Student journalize 20 of the transactions which occur in the third year of 
operations, 

3. Summaries of interviews with employees about operating plan variances 
Students get behind the numbers and figure out what is driving the variances. 

Design Scenario 

This Scenario illustrates how the tools are used to support conceptual and detailed 
design of a BusSim apphcation. Figure 22 illustrates the steps of the first scenario 
in accordance with a preferred embodiment. The designer has gathered requirements 
and determined that to support the cHent's learning objectives, a task is required that 
teaches joumahzation skills. The designer begins the design first by learning about 
journalization herself, and then by using the Knowledge Workbench to sketch a 
hierarchy of the concepts she want the student to learn. At the most general level, 
she creates a root concept of *Joumalization'. She refines this by defining sub- 
concepts of 'Cash related transactions', 'Expense related Transactions', and 
'Expense on account transactions'. These are each fiirther refined to whatever level 
of depth is required to support the quality of the learning and the fidelity of the 
simulation. 

The designer then designs the joumahzation interface. Since a great way to learn is 
by doing, she decides that the student should be asked to JoumaUze a set of 
transactions. She comes up with a set of twenty-two documents that typify those a 
finance professional might see on the job. They include the gamut of Asset, 
Expense, Liability and Equity, and Revenue transactions. Also included are some 
documents that are not supposed to be entered in the journal. These 'Distracters' are 
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included because sometimes errant documents occur in real life. The designer then 
uses the Domain Model features in the Knowledge Workbench to paint a Journal 
An entity is created in the Domain Model to represent each transaction and each 
source document. 

5 Based on the twenty-two documents that the designer chose, she can anticipate 
errors that the student might make. For these errors, she creates topics of feedback 
and populates them with text. She also creates topics of feedback to tell the student 
when they have succeeded. Feedback Topics are created to handle a variety of 
situations that the student may cause. 

10 The next step is to create profiles that the will trigger the topics in the concept tree 
(this task is not computational in nature, so the Transformation Component does not 
need to be configured) . A profile resolves to true when its conditions are met by the 
student's work. Each profile that resolves to true triggers a topic. 
To do some preliminary testing on the design, the designer invokes the Student 

1 5 Simulator Test Workbench. The designer can manipulate the Domain Model as if 
she were the student working in the interface. She drags accounts around to 
different transactions, indicating how she would like them journalized. She also 
enters the dollar amounts that she would like to debit or credit each account. She 
submits her actions to the component engines to see the feedback the student would 

20 get if he had performed the activity in the same way. All of this occurs in the test 
bench without an application interface. 

The last step in this phase is low-fi user testing. A test student interacts with a 
PowerPoint slide or bitmap of the proposed appUcation interface for the 
25 Journalization Task. A facilitator mimics his actions in the test bench and tells him 
what the feedback would be. This simplifies low-fi user testing and helps the 
designer to identify usability issues earher in the design when they are much cheaper 
to resolve. 
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Build Scenario 

Figures 23 and 24 illustrate the steps associated with a build scenario in accordance 
with a preferred embodiment. The instructional designer completes the initial 
5 interaction and interface designs as seen in the previous Scenario. After low-fi user 
testing, the Build Phase begins. Graphic artists use the designs to create the bitmaps 
that will make up the interface. These include bitmaps for the buttons, tabs, and 
transactions, as well as all the other screen widgets. The developer builds the 
interface using the bitmaps and adds the functionahty that notifies the Domain 

10 Model of student actions. Standard event-driven programming techniques are used 
to create code that will react to events in the interface during application execution 
and pass the appropriate information to the Domain Model. The developer does not 
need to have any deep knowledge about the content because she does not have to 
build any logic to support analysis of the student actions or feedback. The developer 

1 5 also codes the logic to rebuild the interface based on changes to the domain model. 

A few passes through these steps will typically be required to get the application 
communicating correctly with the components. The debug utilities and Regression 
Test Workbench streamline the process. After the appUcation interface and 
20 component communication are fimctioning as designed, the task is migrated to 
Usability testing. 

Test Scenario 

This scenario demonstrates the cycle that the team goes through to test the 
25 application. It specifically addresses usability testing, but it is easy to see how the 
tools also benefit fiinctional and cognition testing. Again, we will use the 
Journalization Task as an example. Figure 24 illustrates a test scenario in 
accordance with a preferred embodiment. The test students work through the 
journalization activity. One of the students has made it over half way through the 
30 task and has just attempted to journalize the sixteenth transaction. The student 
submits to the Financial Coach, but the feedback comes back blank. The student 
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notifies the facilitator who right-cUcks on the Financial Coach's face in the feedback 
window. A dialog pops up that shows this is the twenty-seventh submission and 
shows some other details about the submission. The facilitator (or even the student 
in recent efforts) enters a text description of the problem, and fills out some other 
5 fields to indicate the nature and severity of the problem. All the student's work and 
the feedback they got for the twenty-seven submissions is posted to the User 
Acceptance Test (UAT) archive database. 

The instructional designer can review all the student histories in the UAT database 
10 and retrieve the session where the student in question attempted the Joumalization 
Task, The designer then recreates the problem by replaying the student's twenty- 
seven submissions through the component engines using the Regression Test 
Workbench. The designer can then browse through each submission that the student 
made and view the work that the student did on the submission, the feedback the 
15 student got, and the faciUtator comments, if any. Now the designer can use the 

debugging tools to determine the source of the problem. In a few minutes, she is able 
to determine that additional profiles and topics are needed to address the specific 
combinations of mistakes the student made. She uses the Knowledge Workbench to 
design the new profiles and topics. She also adds a placeholder and a script for a 
20 video war story that supports the learning under these circimistances. The designer 
saves the new design of the task and reruns the Regression Test Workbench on the 
student's session with the new task design. After she is satisfied that the new 
profiles, topics, and war stories are giving the desired coverage, she ships the new 
task design file to user testing and it's rolled out to all of the users. 

25 

This example illustrates how a high effort, uncertain process (that once took days) 
can be reduced to a few hours using the BusSim Toolset, Cycle time can be reduced 
dramatically, and complexity, risk and difficulty can be almost eliminated. It shows 
the sharp contrast with the traditional development approach where new designs and 
30 new code can have many unintended consequences that are difficult to test. 
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Execution Scenario: Student Administration 

Figure 25 illustrates how the tool suite supports student administration in accordance 
with a preferred embodiment. When a student first enters a course she performs a 
5 pre-test of his financial skills and fills out an information sheet about his job role, 
level, etc. This information is reported to the Domain Model. The Profiling 
Component analyzes the pre-test, information sheet, and any other data to determine 
the specific learning needs of this student. A curriculum is dynamically configured 
fi-om the Task Library for this student. The application configures its main 
10 navigational interface (if the app has one) to indicate that this student needs to learn 
Journalization, among other things. 

As the student progresses through the course, his performance indicates that his 
proficiency is growing more rapidly in some areas than in others. Based on this 

15 finding, his curriculum is altered to give him additional Tasks that will help him 

master the content he is having trouble with. Also, Tasks may be removed where he 
has demonstrated proficiency. While the student is performing the work in the 
Tasks, every action he takes, the feedback he gets, and any other indicators of 
performance are tracked in the Student Tracking Database. Periodically, part or all 

20 of the tracked data are transmitted to a central location. The data can be used to 
verify that the student completed all of the work, and it can be fixrther analyzed to 
measure his degree of mastery of the content. 

Execution Scenario: Student Interaction 

25 Figure 26 illustrates a suite to support a student interaction in accordance with a 
preferred embodiment. In this task the student is trying to journalize invoices. He 
sees a chart of accounts, an invoice, and the journal entry for each invoice. He 
journalizes a transaction by dragging and dropping an account firom the chart of 
accounts onto the 'Debits' or the 'Credits' line of the journal entry and entering the 

30 dollar amount of the debit or credit. He does this for each transaction. 
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As the student interacts with the interface, all actions are reported to and recorded in 
the Domain Model. The Domain Model has a meta-model describing a transaction, 
its data, and what information a journal entry contains. The actions of the student 
populates the entities in the domain model with the appropriate information. When 
5 the student is ready, he submits the work to a simulated team member for review. 
This submission triggers the Analysis-Interpretation cycle. The Transformation 
Component is invoked and performs additional calculations on the data in the 
Domain Model, perhaps determming that Debits and Credits are unbalanced for a 
given journal entry. 

10 

The Profiling Component can then perform rule-based pattem matching on the 
Domain Model, examining both the student actions and results of any 
Transformation Component analysis. Some of the profiles fire as they identify the 
mistakes and correct answers the student has given. Any profiles that fire activate 

15 topics in the Remediation Component. After the Profihng Component completes, 
the Remediation Component is invoked. The remediation algorithm searches the 
active topics in the tree of concepts to determine the best set of topics to deliver. 
This set may contain text, video, audio, URLs, even actions that manipulate the 
Domain Model. It is then assembled into prose-like paragraphs of text and media 

20 and presented to the student. The text feedback helps the student localize his 
joumahzation errors and understand why they are wrong and what is needed to 
correct the mistakes. The student is presented with the opportunity to view a video 
war story about the tax and legal consequences that arise fi-om incorrect 
joumahzation. He is also presented with links to the reference materials that 

25 describe the fimdamentals of journalization. 

The Analysis-Interpretation cycle ends when any coach items that result in updates 
to the Domain Model have been posted and the interface is redrawn to represent the 
new domain data. In this case, the designer chose to highlight with a red check the 
30 transactions that the student journalized incorrectly. 
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IIL The Functional Definition of the ICAT 

This section describes the feedback processes in accordance with a preferred 
embodiment. For each process, there is a definition of the process and a high-level 
description of the knowledge model. This definition is intended to give the reader a 
baseline understanding of some of the key components/objects in the model, so that 
he can proceed with the remaining sections of this paper. Refer to the Detailed 
Components of the ICAT for a more detailed description of each of the components 
within each knowledge model. To gain a general understanding of the ICAT, read 
only the general descriptions. To understand the ICAT deeply, read this section and 
the detailed component section regarding knowledge models and algorithms. These 
processes and algorithms embody the feedback model in the ICAT. There are six 
main processes in the ICAT, described below and in more detail on the following 
pages. 

Remediation Process Diagram 

Figure 27 illustrates the remediation process in accordance with a preferred 
embodiment. Remediation starts as students interact with the application's interface 
(process #1). As the student tries to complete the business deliverable, the 
application sends messages to the ICAT about each action taken (process #2). When 
the student is done and submits work for review, the ICAT compares how the 
student completed the activity with how the designer stated the activity should be 
completed (this is called domain knowledge). From this comparison, the ICAT get a 
count of how many items are right, wrong or irrelevant (process #3), With the count 
complete, the ICAT tries to fire all rules (process #4). Any rules which fire activate 
a coach topic (process #5), The feedback algorithm selects pieces of feedback to 
show and composes them into coherent paragraphs of text (process #6). Finally, as 
part of creating feedback text paragraphs, the ICAT replaces all variables in the 
feedback with specifics fi"om the student's work. This gives the feedback even more 
specificity, so that it is truly customized to each student's actions. 
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L Student interacts with interface to create business deliverable 

Description 

The student completes the deliverables of the Task by interacting with the interface 
5 objects. These actions may be button chcks, dragging of text, selection of items 
from a list, etc. An example is the Journalization task shown below. Figure 28 
illustrates a display of journalization transactions in accordance with a preferred 
embodiment. To interact with the display, the student must journalize the twenty- 
four transactions presented. To journalize a transaction, the student clicks the "next" 

10 and "previous" buttons to move between transactions. Once at a transaction, the 
student cKcks and drags an account name from the chart of accounts-which is split 
into Assets, Liabilities, Revenues and Expenses-onto the debit or credit side of the 
journal entry. Once the joumal entry has been made, the student must type in how 
much to debit or credit. Each one of these buttons, draggable items, and text fields 

15 are interface objects which can be manipulated. 

Knowledge Model 
Interface Objects 

In any GBS Task, the student must manipulate controls on the application interface 
20 to complete the required deliverables. Figure 29 illustrates the objects for the 
journalization task in accordance with a preferred embodiment. 

The following abstract objects are used to model all the various types of interface 
interactions. 
25 Sourceltem 

A Sourceltem is an object the student uses to complete a task. In the journalization 
example, the student makes a debit and credit for each transaction. The student has a 
finite set of accounts with which to respond for each transaction. Each account that 
appears in the interface has a corresponding Sourceltem object. In other words, the 
30 items the student can manipulate to complete the task (account names) are called 
Sourceltems. 
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Source 

A Source is an object that groups a set of Sourceltem objects together. Source 
objects have a One-To-Many relationship with Sourceltem objects. In the 
5 journalization example, there are four types of accounts: Assets, Liabilities and 
Equity, Revenues, and Expenses. Each Account is of one and only one of these 
types and thus appears only under the appropriate tab. For each of the Account type 
tabs, there is a corresponding Source Object. 



10 Target 

A Target is a fixed place where students place Sourceltems to complete a task. In 
the joimiaUzation example, the student places accounts on two possible targets: 
debits and credits. The top two lines of the journal entry control are Debit targets 
and the bottom two lines are Credit targets. These two targets are specific to the 
1 5 twelfth transaction. 

TargetPage 

A TargetPage is an object that groups a set of Target objects together. TargetPage 
objects have a One-To-Many relationship with Target objects (just Hke the Source to 
20 Sourceltem relationship). In the journalization example, there is one journal entry 
for each of the twenty-two transactions. For each journal entry there is a 
corresponding TargetPage object that contains the Debits Target and Credits Target 
for that journal entry. 

25 2. Reporting student actions to the ICA T 
Description 

As the student manipulates the application interface, each action is reported to the 
ICAT, In order to tell the ICAT what actions were taken, the application calls to a 
database and asks for a specific interface control's ID. When the application has the 
30 ID of the target control and the Sourceltem control, the application notifies the ICAT 
about the Target to Sourceltem mapping. In other words, every time a student 
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manipulates a source item and associates it with a target (e.g,, dragging an account 
name to a debit line in the journal), the user action is recorded as a mapping of the 
source item to the target. This mapping is called a UserSourceltemTarget. Figure 
30 illustrates the mapping of a source item to a target item in accordance with a 
5 preferred embodiment, 

3. Student submits deliverables to one team member 
Description 

When the student is ready, he submits his work to one of the simulated team 
members by clicking on the team member's icon. When the ICAT receives the 
student's work, it calculates how much of the work is correct by concept. Concepts 
in our journalization activity will include Debits, Credits, Asset Accounts, etc. For 
each of these concepts, the ICAT will review all student actions and determine how 
many of the student actions were correct. In order for the ICAT to understand which 
targets on the interface are associated with each concept, the targets are bundled into 
target groups and prioritized in a hierarchy. Figure 31 illustrates target group 
bundles in accordance with a preferred embodiment. For each target group~or 
concept, such as debit-a number of aggregate values will be calculated. These 
aggregate values determine how many student actions were right, wrong or 
irrelevant. 

Knowledge Model 
TargetGroup 

A TargetGroup object represents a concept being learned. It is a group of Target 
25 objects related on a conceptual level. The TargetGroup objects in a Task are 

arranged in a hierarchy related to the hierarchy of concepts the student must learn. 
By analyzing the student's responses to the Targets in a TargetGroup, the ICAT can 
determine how well a student knows the concept. By utilizing the conceptual 
hierarchy of TargetGroups the ICAT can determine the most appropriate 
30 remediation to deliver to help the student understand the concepts. 
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TargetGroup Hierarchy 

The TargetGroup objects in a Task are arranged in a hierarchical tree structure to 
model the varying specificity of concepts and sub-concepts being learned in the 
Task. The designer defines the parent-child relationships between the TargetGroups 
5 to mimic the relationships of the real world concepts. This hierarchy is used in the 
determination of the most appropriate feedback to deUver. Concepts that are higher 
(more parent-like) in the TargetGroup structure are remediated before concepts that 
are modeled lower (children, grandchildren, etc.) in the tree. Figure 32 illustrates a 
TargetGroup Hierarchy in accordance with a preferred embodiment. 

10 

In the joumalization example, the main concept being taught is joumalization. The 
concept of joumahzation can be divided into more specific sub-concepts, for 
example joumalizing cash-for-expense transactions and journalizing expense-on- 
account transactions. These may further be divided as necessary. The designer 

1 5 teaches this conceptual knowledge to the IC AT by creating a TargetGroup called 
"Joumalizing Transactions" with two child TargetGroups "Joumahzing Cash for 
Expense Transactions" and "Joumalizing Expense On Account Transactions". The 
top-most TargetGroup in the Task, "Joumalizing Transactions" contains all of the 
transactions in the Task. Child target groups will include just the first three 

20 transactions and transactions four to twenty. 

Therefore when the when the ICAT determines how much of the task is correct, it 
will calculate values for the first three journal entries and the next sixteen. 
Calculating these two separate numbers will allow the ICAT to provide specific 

25 feedback about the first three and separate feedback about the next sixteen 

transactions. Here is a section of the target group hierarchy for the journalize task. 
Figure 33 illustrates a small section the amount of feedback in accordance with a 
preferred embodiment. By analyzing the responses to the targets in the each of the 
targetgroups, we can determine how many of the transactions the student has 

30 attempted, whether mistakes were made, what the mistakes were, etc. We can then 
assemble feedback that is very specific to the way the student completed the 
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deliverables. By analyzing the student's responses to a group of conceptually related 
requests, we can determine the degree of success with which the student is learning 
the concept, 

5 4. ICAT analyzes deliverables with Rules 
Description 

After the ICAT has calculated the aggregate values for the student's deliverables, it 
analyzes the deUverables by attempting to fire all of the Rules for that task. Rules 
that can fire, activate CoachTopics. Figure 34 illustrates an analysis of rules in 
1 0 accordance with a preferred embodiment. 

5. Select appropriate remediation coach topics 
Description 

Once all possible coach topics are activated, a feedback selection analyzes the active 
1 5 pieces of remediation within the concept hierarchy and selects the most appropriate 
for delivery. The selected pieces of feedback are then assembled into a cohesive 
paragraph of feedback and delivered to the student. Figure 35 illustrates a feedback 
selection in accordance with a preferred embodiment. 

20 Feedback Selection Algorithm 

After the ICAT has activated CoachTopics via Rule firings, the Feedback Selection 
Algorithm is used to determine the most appropriate set of Coachltems (specific 
pieces of feedback text associated with a CoachTopic) to deliver. The Algorithm 
accomplishes this by analyzing the concept hierarchy (TargetGroup tree), the active 

25 CoachTopics, and the usage history of the Coachltems. Figure 36 is a flowchart of 
the feedback logic in accordance with a preferred embodiment. There are five main 
areas to the feedback logic which execute sequentially as listed below. First, the 
algorithm looks through the target groups and looks for the top-most target group 
with an active coach topic in it. Second, the algorithm then looks to see if that top- 

30 most coach item is praise feedback. If it is praise feedback, then the student has 

correctly completed the business deliverable and the ICAT will stop and return that 
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coach item. Third, if the feedback is not Praise, then the ICAT will look to see if it 
is redirect, polish, mastermind or incomplete-stop. If it is any of these, then the 
algorithm will stop and retum that feedback to the user. Fourth, if the feedback is 
focus, then the algorithm looks to the children target groups and groups any active 
5 feedback in these target groups with the focus group header. Fifth, once the 
feedback has been gathered, then the substitution language is run which replaces 
substitution variables with the proper names. 

Once the ICAT has chosen the pieces of feedback to retum, the feedback pieces are 
10 assembled into a paragraph. With the paragraph assembled, the ICAT goes through 
and replaces all variables. There are specific variables for Sourceltems and Targets. 
Variables give feedback specificity. The feedback can point out which wrong 
Sourceltems were placed on which Targets. It also provides hints by providing one 
or two Sourceltems which are mapped to the Target. 

15 

IV. The ICAT Development Methodology for creating Feedback 
The Steps Involved in Creating Feedback 

The goal of feedback is to help a student complete a business deliverable. The tutor 
needs to identify which concepts the student understands and which he does not. 
20 The tutor needs to tell the student about his problems and help him understand the 
concepts. 

There are seven major steps involved in developing feedback for an application. 

25 First, creating a strategy — The designer defines what the student should know. 
Second, limit errors through interface — The designer determines if the interface 
will identify some low level mistakes. 

Third, creating a target group hierarchy — The designer represents that knowledge 
in the tutor. 

30 Fourth, sequencing the target group hierarchy — The designer tells the tutor which 
concepts should be diagnosed first. 
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Fifth, writing feedback — The designer writes feedback which tells the student how 
he did and what to do next. 

Sixth, writing Levels of Feedback — The designer writes different levels of 
feedback in case the student makes the same mistake more than once. 
5 Seventh, writing rules — The designer defines patterns which fire the feedback. 

Creating a Feedback Strategy 

A feedback strategy is a loose set of questions which guide the designer as he creates 
rules and feedback. The strategy describes what the student should learn, how he 
10 will try and create the business deliverable and how an expert completes the 

deliverable. The goal of the appUcation should be for the student to transition firom 
the novice model to the expert model. 

What should the student know after using the application? 

15 The first task a designer needs to complete is to define exactly what knowledge a 
student must leam by the end of the interaction. Should the student know specific 
pieces of knowledge, such as formulas? Or, should the student understand high 
level strategies and detailed business processes? This knowledge is the foundation of 
the feedback strategy. The tutor needs to identify if the student has used the 

20 knowledge correctly, or if there were mistakes. An example is the journal task. For 
this activity, students need to know the purpose of the joumaUzing activity, the 
specific accounts to debit/credit, and how much to debit/credit. A student's 
debit/credit is not correct or incorrect in isolation, but correct and incorrect in 
connection with the dollars debited/credited. 

25 

Because there are two different types of knowledge-accounts to debit/credit and 
amounts to debit/credit--the feedback needs to identify and provide appropriate 
feedback for both types of mistakes. 

30 How will a novice try and complete the task? 

Designers should start by defining how they beUeve a novice will try and complete 
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the task. Which areas are hard and which are easy for the student. This novice view 
is the mental model a student will bring to the task and the feedback should help the 
student move to an expert view. Designers should pay special attention to 
characteristic mistakes they beheve the student will make. Designers will want to 
5 create specific feedback for these mistakes. An example is mixing up expense 
accounts in the journal activity. Because students may mix up some of these 
accounts^ the designer may need to write special feedback to help clear up any 
confusion. 

10 How does an expert complete the task? 

This is the expert model of completing the task. The feedback should help students 
transition to this understanding of the domain. When creating feedback, a designer 
should incorporate key features of the expert model into the praise feedback he 
writes. When a student completes portion of the task, positive reinforcement should 

15 be provided which confirms to the student that he is doing the task correctly and can 
use the same process to complete the other tasks. 

These four questions are not an outline for creating feedback, but they define what 
the feedback and the whole application needs to accomplish. The designer should 
20 make sure that the feedback evaluates all of the knowledge a student should leam. 
In addition, the feedback should be able to remediate any characteristic mistakes the 
designer feels the student will make. Finally, the designer should group feedback so 
that it retums feedback as if it were an expert. With these components identified, a 
designer is ready to start creating target group hierarchies. 

25 

Limit Errors Through Interface 

When the designer defines a feedback strategy, the designer defines the skills he 
wants the student to leam and the mistakes he thinks the student will make. Not all 
of the mistakes need to be corrected with ICAT generated feedback, some can be 
30 limited with or remediated through the interface. Limiting mistakes with the 
interface simply means that the system pops-up a message as the student works, 
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identifying a mistake. An example interface corrected error is in the journalization 
activity when the interface points out that debits do not equal credits. Here, this is a 
low level mistake which is more appropriate to remediate through the interface than 
through the ICAT. The application simply check to see if the debit number equal the 
5 credit number and if they do not then the system message is returned. Figure 37 
illustrates an example of separating out some mistakes for the interface to catch and 
others for the ICAT to catch has positive and negative impacts in accordance with a 
preferred embodiment. 

10 Positive 

The most obvious reason for eliminating mistakes through the interface is that can 
be easier for the designer and developer to catch them at this level than to leave them 
for the ICAT. 

15 Negative 

The reason to avoid interface-driven feedback is that it splinters the feedback 
approach which can make the job of creating a coherent feedback approach more 
difficult. 

20 Because there are positive and negative repercussions, designers need to select the 
when to remediate through the interface carefully. The criteria for making the 
decision is if the mistake is a low level data entry mistake or a high level intellectual 
mistake. If the mistake is a low level mistake, such as miss-typing data, it may be 
appropriate to remediate via the interface. If the designer decides to have the 

25 interface point out the mistakes, it should look as if the system generated the 

message. System generated messages are mechanical checks, requiring no complex 
reasoning. In contrast, complex reasoning, such as why a student chose a certain 
type of account to credit or debit should be remediated through the ICAT. 

30 System messages 

It is very important that the student know what type of remediation he is going to get 
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from each source of information. Interface based remediation should look and feel 
like system messages. They should use a different interface from the ICAT 
remediation and should have a different feel. In the journalization task described 
throughout this paper, there is a system message which states "Credits do not equal 
5 debits." This message is delivered through a different interface and the blunt short 
sentence is unlike all other remediation. 

The motivation for this is that low level data entry mistakes do not show 
misunderstanding but instead sloppy work. Sloppy-work mistakes do not require a 

10 great deal of reasoning about why they occurred instead, they simply need to be 
identified. High-level reasoning mistakes, however, do require a great deal of 
reasoning about why they occurred, and the ICAT provides tools, such as target 
groups, to help with complex reasoning. Target group hierarchies allow designers to 
group mistakes and concepts together and ensure that they are remediated at the 

15 most appropriate time (i.e., Hard concepts will be remediated before easy concepts). 
Timing and other types of human-like remediation require the ICAT; other low-level 
mistakes which do not require much reasoning include: 

Incomplete 

20 If the task requires a number of inputs, the interface can check that they have all 

been entered before allowing the student to proceed. By catching empty fields early 
in the process, the student may be saved the frustration of having to look through 
each entry to try and find the empty one. 

25 Empty 

A simple check for the system is to look and see if anything has been selected or 
entered. If nothing has been selected, it may be appropriate for the system to 
generate a message stating "You must complete X before proceeding". 



30 



Numbers not matching 

Another quick check is matching numbers. As in the joumalization activity, is often 
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useful to put a quick interface check in place to make sure numbers which must 
match do. Small data entry mistakes are often better remediated at the interface 
level than at the tutor or coach level (when they are not critical to the learning 
objectives of the course). 

5 

There are two main issues which must be remembered when using the interface to 
remediate errors. First, make sxire the interface is remediating low level data entry 
errors. Second, make sure the feedback looks and feels different from the ICAT 
feedback. The interface feedback should look and feel like it is generated from the 
1 0 system while the ICAT feedback must look as if it were generated from an 
intelhgent coach who is watching over the student as he works. 

Creating the Target Group Hierarchy 

Target groups are sets of targets which are evaluated as one. Returning to the 

1 5 severity principle of the feedback theory, it is clear that the tutor needs to identify 
how much of the activity the student does not understand. Is it a global problem and 
the student does not understand anything about the activity? Or, is it a local problem 
and the student simply is confiised over one concept? Using the feedback algorithm 
described earlier, the tutor will return the highest target group in which there is 

20 feedback. This algorithm requires that the designer start with large target groups and 
make sub-groups which are children of the larger groups. The ICAT allows students 
to group targets in more than one category. Therefore a debit target for transaction 
thirteen can be in a target group for transaction thirteen entries as well as a target 
group about debits and a target group which includes all source documents. Target 

25 should be grouped with four key ideas in mind. Target groups are grouped 
according to: 
Concepts taught 
Interface constraints 
Avoidance of information overload 

30 Positive reinforcement 
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The most important issue when creating target groups is to create them along the 
concepts students need to know to achieve the goal Grouping targets into groups 
which are analogous to the concepts a student needs to know, allows the tutor to 
review the concepts and see which concepts confuse the student. As a first step, a 
5 designer should identify in an unstructured manner all of the concepts in the domain. 
This first pass will be a large list which includes concepts at a variety of 
granularities, fi:om small specific concepts to broad general concepts. These 
concepts are most likely directly related to the learning objectives of the course. 

10 With all of the concepts defined, designers need to identify all of the targets which 
are in each target group. Some targets will be in more than one target group. When 
a target is in more than one target group, it means that there is some type of 
relationship such as a child relationship or a part to whole relationship. The point is 
not to create a structured Ust of concepts but a comprehensive list. Structuring them 

1 5 into a hierarchy will be the second step of the process. 

In the journalization activity, the largest concept is the recording a transaction. 
Other important ideas are debits and credits. Debit and credit targets, however, are 
included in the overall transaction target group which means that it is either a part- 
20 whole relationship or a child relationship. Figure 38 is a block diagram of the 

hierarchical relationship of a transaction in accordance with a preferred embodiment. 



Concepts Taught: Part-whole Concepts 

With all of the target groups laid out, the designer needs to identify the relationships 
25 between concepts. One type ofrelationship is the part-whole relationship. Part- 
whole relationships—as the name denotes-identified which sub-components make 
up larger concepts. Identifying these relationships is important because the tutor 
will want to see if the student does not understand the whole concept or just one 
part. If there are no major errors in the concept as a whole, then the tutor will look 
30 to see if the student made any major errors in one part of the concept. 
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Example: 

In the journalizing activity, there will be a target group called transaction. In 
5 transaction, there are two parts: debits and credits. When the tutor reviews the 
student's work, if there are no problems with the target group transactions, then the 
tutor will go to the next level and look for errors in the target group debits and 
credits. Because debits and credits are included in an overall transaction, there is a 
part-whole relationship to the concept transaction. 

10 

Concept Taught: Child Concepts 

In addition to part-whole relationships, designers need to identify child-parent 
relationships. In contrast to part-whole relationships, child-parent relationships 
define instances of abstract concepts. An example is "The dictionary is a book". 
15 "Dictionary" is a child concept to "book". The "dictionary" concept has all of the 
attributes of the "book" concept, and it is an instance of the concept which means 
that it contains extra attributes. Students may understand the concept in general but 
may be confused about a particular instance. 

20 Example: 

In the joumahzation activity, the concept transaction can be broken down into two 
sections: the debit and the credit. And each of those can be specialized into 
specialization categories, such as a credit to "Accounts payable". Students may not 
be confused about debits but the mstance "Accounts Payable". 

25 

Interface Constraints 

Interface Constraint: Business Deliverable 

When creating target group hierarchies, designers need to consider the type of 
deliverable the student is creating. For each of the sections of the deliverable, the 
30 designer needs to create a target group. The target groups should contain an orderly 
structure, such as moving firom top to bottom. Reviewing the deliverable in the 



-92- 



order it is created structures the critique so that students know where to look next, 
after receiving feedback. In the current InteUigent Tutoring Agent, this structuring 
of feedback around the student-created deUverable can be accompHshed in two 
ways. First, the designer can make every section of the dehverable a target. In 
5 addition, the designer can make some sections targets and some modifying 

attributes. Modifying attributes can be remediated on specifically, or in conjunction 
with the target. 

In the joumalization activity, the sections of the product~the joumal entry-mirrors 
10 the concepts involved—debits and credits. But there are a few extra items on the 
joumal which are (in most cases) not involved in the main concepts being taught, 
and these are the dollar amounts to be journalized. The dollar amounts which are 
joumaUzed are associated with the joumal entry as an attribute. Attributes modify 
the source item (account name), which makes it possible to tell if the source item is 
15 correct alone or with the attribute attached. As a designer, feedback should be 

created which takes all of this into account. Students should be told if they have the 
joumal entry correct and the amount wrong, or if they have the whole thing wrong. 

Interface Constraint: Screen Space 

20 Many times one concept will span many sections of the interface. It is important to 
group the target groups so that they are interface specific. Therefore, even though 
one product may span multiple interfaces, the target groups should be centered 
around the interfaces so that the students receive feedback only about what they can 
see. 

25 

In the joumahzation activity, the sections of the deliverable-the collection of joumal 
entries in the ledger ™ span many separate interfaces. Each source document must 
be seen individually. Therefore, some target groups are organized across all source 
documents — such as all debits - and others are specific to the individual source 
30 documents ™ such as that source document's debits. The target group's hierarchy 
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must include a section for across source documents - across interfaces - and those 
within one source document — one interface. 

Information Overload 
5 As with any real-life tutor, you do not want to give too much information for the 
student to digest at once. If there are twenty-five problems, the tutor should not give 
feedback about all errors simultaneously. Instead, the tutor should give feedback 
about just two or three things which the student can correct before asking for more 
feedback. 

10 

In the joumaUzation activity, there are a limited number of targets on the interface at 
one time-one debit and one credit. But if it were the whole General Ledger, it could 
have too many pieces of feedback for the student to digest at once and could 
overwhelm the student. In this case, the designer should scale the feedback so that 
15 just a handful come back at once. This is best done by having small target groups 
defined, but can also be done by identifying to the tutor how many different pieces 
of remediation are appropriate to deliver at one time. 

Positive Reinforcement 
20 In addition to creating target groups which are small in size, designers may want to 
create target groups which evaluate the first few steps a student makes. These early 
target groups will allow the student to see if he is on track and understand the goal of 
the interaction. This is, in general, a good remediation strategy, but may not be 
relevant in all learning situations. 

25 

In the joumalization activity, there are twenty source documents to joumalize. 
Students should NOT be encouraged to ask for feedback at every step, but when they 
have completed all of their work. This will ensure that students try and learn all of 
the information first and not rely completely on the hints of the tutor. But, target 
30 groups defined for just the first three entries allow for feedback and hints to be 
provided at the onset of the task, diminishing once these entries are correct. 
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Sequencing the Target Group Hierarchy 

For feedback to be as effective as possible, it needs to provide the right information 
at the right time. If feedback is given too early, it is confusing; if feedback is given 
5 too late it is frustrating. In the ICAT, feedback is retumed according to Target 
Groups, The tutor will look at the highest target group, if there is no feedback in 
that target group, the tutor will look at the children target groups in order of priority. 

Figure 39 is a block diagram illustrating the feedback hierarchy in accordance with a 
10 preferred embodiment. In Figure 39, the tutor will first look for any relevant 

feedback to be delivered in target group #1 A. If there is nothing there, then the tutor 
will look in the highest prioritized child target group — in the B tier. If there is 
nothing in that target group, then the tutor will look in the highest child target group 
of target group #1B which is target group #1C. Because the target group priority 
15 determines where the tutor looks for feedback within tier, a great deal of thought 
needs to be given to what comprises a target group and how they are structured. 
There are four guiding principles which will help structure target groups to provide 
the right information at the right time and help the student make the most of the 
information provided. 

20 

Positive Reinforcement First 

Designers should identify the first few components a student will try and complete 
first and sequences them first. This target group will evaluate just the first few 
moves a student makes and will tell him how he is doing and how to apply the 
25 knowledge gained from the first few steps to the rest of the work he has to do. 

In the joumahzation activity, students need to have reinforcement that they are on 
the right track before trying all of the joumal entries. Therefore, the first three are 
grouped together and students can feedback on how they completed this sub-group 
30 before having to complete the rest. Completing this subsection gives students the 
positive reinforcement they need to complete the rest. 
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Easy Before Hard 

If all of the target groups are of equivalent size, designers need to sequence easier 
concepts before more complicated concepts. By placing easier concepts first, a 
5 student will gain confidence in their understanding of the domain and in their ability 
to complete the deUverable. In addition, most complicated concepts are built on 
easier ones so that presenting easier concepts first will allow the student to gain the 
experience they need to complete the most comphcated concepts. In the 
journalization activity, two legged journal entries are inherently easier than three 
10 legged and four legged journal entries. Therefore when a designer must sequence 
target groups of equal size, the designer should sequence the two legged journal 
entries before the three and four legged entries. 

First Things First 

15 Besides sequencing easier concepts before hard concepts, another strategy is to 
sequence target groups in order that they need to be completed. If completing one 
section of the deliverable is a prerequisite for completing another section of the 
deUverable, it makes sense to sequence those targets first. In the joumaUzation 
activity, a source document needs to be joumalized in terms of the account name and 

20 in terms of the dollar amount. However, the account name must be identified before 
the amount is entered. It makes no difference whether the dollar figure of the 
account is right or wrong, until the student has the correct account name. 

Writing Feedback 

25 Creating and structuring target group hierarchies determines what is evaluated and 
the order the feedback is returned. Once the hierarchy has been created and 
structured, designers need to write feedback which will help the student complete his 
goal. Going back to the goals of the tutor as educator, feedback needs to accomplish 
the following goals: 

30 Identify concepts students do not understand 
Identify student mistakes 
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Prompt students to reflect on their mistakes 
Reinforce correct concepts and ideas 



These goals can be thought of in two sections. The first two are evaluative and the 
5 second two are instructive . Evaluation and instruction are two of the three main 
components of a piece of feedback text. The third component is Scope. These three 
components are described in more detailed below, beginning with Scope, as it is 
generally the first portion of a piece of feedback text. 

10 What the Feedback is Evaluating (Scoge) 

The most important information feedback provides a student is what the tutor is 
reviewing. In most instances, the student will have completed lots of different 
actions before asking the tutor to review his work. Because the student has 
completed a lot of different actions, the tutor first needs to describe what portion of 

15 the activity or deliverable is being reviewed. There are generally three ways to 
scope what the tutor is reviewing. 

All work 

The tutor is looking at everything the student did. Some instances when feedback 
20 should look at everything the student has done are praise level feedback and redirect 
level feedback. I looked at all of the journal entries and there are problems in 
many of them. Why don't you.... 

A localized area of work 
25 The tutor is looking at a subset of work the student completed. The greatest use of 
localized scoping if focus feedback. The feedback is focusing the student on one 
area of difficulty and asking him to correct it. I am looking at the first five journal 
entries you made, and here are the first three problems I found. The first. „ 



30 



A specific problem or error 

The tutor is focusing on one error and/or problem and helping the student understand 
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that error. Specific problem scoping is good for classic mistakes a student may 
make and the designer may want to remediate. In the first journal entry, you 
incorrectly debited Accounts Payable. Review that transaction... 

How the Student Did (Evaluation ^ 

The second section of the feedback text should describe how the student did. This is 
where the severity principle is applied and the feedback is either redirect, focus, 
polish or praise. 

Redirect 

Redirect feedback is appropriate for very severe errors: severe mistake sand 
misconceptions. This degree of severity can be assessed aggregately by recognizing 
there are problems throughout the student's work or it can be done specifically by 
recognizing some basic items are incorrect. 

Example: 

I am looking at the first five joumal entries you made, and there are problems in 
most of them. Why don't you... I am looking at the first five joumal entries you 
made, and you have made some basic mistakes with debits and credits. Why 
don't you... 

Focus 

Focus feedback is appropriate for localized mistakes or misconceptions. Focus level 
mistakes can be identified aggregately by identifying an area in which there are a 
number of mistakes or specifically by identifying that some of the building block 
ideas are wrong. 

Example: 

I am looking at the first five joumal entries you made, and there are problems in 
many of the debits. Why don't you... 
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I am looking at the first five journal entries you made, I see problems when 
transactions are "on account". Why don't you,.. 

5 Polish 

Polish level feedback is for syntactic problems. Student understand the main ideas 
and have no local problems. There may be just one or two mistakes the student has 
made. Polish feedback should specify where the mistake is. 

10 Example: 

I am looking at the first five joumal entries you made, and the third journal entry 
has the debit incorrect. Why don't you... 

Praise 

1 5 Praise level feedback is reserved for instances of "correctness"; the deliverable is 
correct and ready to be used in the business. 



20 

Example: 

I am looking at the first five joumal entries you made, and they are all correct, 
remember... 

25 

Mastermind 

Mastermind feedback is reserved for instances where the student is not trying to 
learn a topic but trying to cheat his way through by repeatedly asking for feedback. 
The feedback needs to be written so that the student recognizes that the tutor wants 
30 more work completed before providing feedback. 
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Example: 

You have not changed much of your work since the last time you asked me to 

review it. Review... 

Incomplete 

5 Incomplete feedback is reseived for instances where the student has not completed 
all of the required work. It should be remembered that sometimes it is desired to 
give substantive feedback before everything is complete so the student learns the 
process and concepts before trying to complete the whole deliverable. 

10 Example: 

You have not done all of your work. I would like you to try completing all 
journal entries before asking for my review. 

What the Student Should Do Next (Instruction) 

15 The final piece of information the student needs is what to do next. The student 
knows what the tutor reviewed and knows how he performed. The only thing the 
student does not know is what to do next. The type of instruction does not have to 
correspond with the severity of the error. The instructions can be mixed and 
matched with the type of error. Some of the actions a student could be asked to 

20 perform are as follows. 

Review the general concept 

If the tutor recognizes that there are many errors throughout the deliverable, the tutor 
may suggest that the student go through a review of the supporting materials 
25 provided to gain an understanding of the ideas and skills needed to complete the 
task. 

Example: 

There are problems in many joumal entries, why don't you review how to 
30 journalize transactions and then review your journal entries. 
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Review a section of the student's work 

If the student has many errors in one section, then the tutor may suggest that the 
student go and review that section of their work, 

5 Example: 

There are problems in the first five journal entries, why don't you review them. 
Review work with a hint 

If there is a certain idea or concept which the tutor believes the student does not 
1 0 understand, then the tutor may give a hint in the form of a question or statement for 
the student to think about before trying to fix the problems. 

Example: 

There are problems in the first five joumal entries. It looks like you have made 
15 some errors with the expense debits. Remember that expenses are not 

capitalized. Why don't you review the first five journal entries looking for 
journal entries which contained incorrect debits to expense accounts. 

Review work looking for type of error 
20 If there is a specific type of error that the student has made throughout his work, then 
the tutor may tell the student the specific type of error and ask him to go through his 
work correcting this error. 

Example: 

25 There are problems in the first five joumal entries. You have switched all of your 
journal entries on account debits. Why don't you go and fix them. 

Review work looking for specific error 

If there is a specific error that the student has committed, the tutor may tell the 
30 student the specific error committed and where the error is. 
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Example: 

There is a problem with your third journal entry. The debit should not be 
"Accounts Payable." 

5 Review work because it is correct and the student will want to use this analysis 
technique in the future. 

Example: 

Your first three journal entries are correct. Remember that the major distinction 
10 between paying for something "On Account" or in cash. This is a distinction 
you will need to make in the future. 

Do more work 

If it can be determined that the student is simply asking for feedback to "Cheat" his 
15 way through the course, feedback should be provided to tell the student that he needs 
to try and correct many more entries before receiving substantive feedback. 

Example: 

You have not changed much of your work since the last time you asked me to 
20 review it. Please review all of your journal entries and correct many of them. 

Complete your work 

When it can be determined that all of the work which should be complete is not, the 
25 feedback needs to tell the student to complete the work required. 

Example: 

You have not completed all of your work. I would like you to try completing all 
journal entries before asking for my review. 

30 
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Writing Levels of Feedback 

Even with effective feedback, students will often make the same types of mistakes 
again or in different situations. The question is what to tell the student the second 
5 time he makes the same or similar mistakes. We assume that telling the student the 
same thing over and over is not the right answer. Therefore instead of telling the 
student the same thing, the feedback cycles to a lower, or secondary, level. At this 
time, we beheve that three levels of feedback is appropriate for most instances. If 
the target group is particularly complex, however, additional levels of feedback may 
10 be required. 

First Level of Feedback 

The first level of feedback should focus more on teUing the student what is wrong 
and letting the student try and figure it out on his own. Therefore using the 
15 paradigm described above, the student should be told what the tutor is reviewing, 
how he did and asked to retry it or referred to some reference which could be used to 
find the answer. 

20 Example: 

There are problems in many journal entries. Why don't you review how to 
journalize transactions and then review your work. 

Second Level of Feedback 
25 The second level of feedback should give hints and provide pieces of the puzzle. I 
can be assumed that students cannot figure out the problem on their own and need 
some help. It is appropriate at this point to ask the student to review their work with 
a specific hint in mind or with a question to think about. Also, if there are specific 
points in the reference system to review, this is the time to provide them. 



30 
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Example: 

There are problems in the first five journal entries. It looks like you have made 
some errors with expense debits. Remember that expenses are not capitalized. Why 
don't you review the first five journal entries looking for journal entries which 
contain incorrect debits to expense accounts. 

Third Level of Feedback 

The third level of feedback is appropriate for examples. Use the parameter 
substitution language to insert an example of an error they made into the feedback. 
Walk the student through the thought process he should use to solve the problem and 
provide and example of how they did the work right and how they did the work 
wrong. 

Example: 

There are problems in many of your joumal entries. It looks hke you have made 
some errors distinguishing between "on account" and "cash" credits. In particular, 
you characterized joumal entry #12 as a cash purchase when in fact it is an "on 
account" purchase. Remember bills which are not paid immediately are paid on 
account. 

Writing Rules 

With the hierarchies created and sequenced and the feedback written, the designer is 
ready to write rules. Rules fire the particular pieces of feedback the student reads. 
To write effective rules, designers must realize the piece of feedback and the rule are 
one and the same. The only difference is the language used. The feedback is written 
in English and the rules are written as patterns. 

Example rule: 

If the student has attempted all of the first three joumal entries 
And they all contain at least one mistake 
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Then provide feedback "In the first three journal entries you have made at least one 
mistake in each. Why don't you review them and see if you can find the mistakes." 

In the above example, the rules has two conditions (attempt all three joumal entries 
5 and have at least one mistake in each). The feedback is an explicit statement of that 
rule. The feedback states "In the first three joumal entries you have made at least 
one mistake in each. Why don't you review them and see if you can find any 
mistakes." 

10 The rule and the feedback are exactly the same. Keeping the rules and the feedback 
tightly linked ensures that the student receives the highest quality feedback. The 
feedback exactly explains the problem the rules found. If the feedback is more 
vague than the rule, then the students will not understand the exact nature of the 
problem. The feedback will simply hint at it. If the feedback is more specific than 

1 5 the rule, students will become confused. The student may not have made the 
specific error the feedback is referring to under the umbrella rule. 

Types of Rules 

Because the rules need to map to the feedback, there will be six types of rules 
20 associated with the six types of feedback: Praise, Polish, Focus, and Redirect, along 
with Mastermind and Incomplete. 

Praise 

Praise rules need to look for one hundred percent correct and NO errors. If the rule 
25 does not explicitly look for no errors, the rule will also fire when the student has all 
of the right answers but also some of the wrong ones. 

If 100% of the targets in the first three joumal entries are correct 
3 0 And they all contain no mistakes 
Then provide praise feedback 
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Praise rules can be applied in many places other than the highest task level. Praise 
rules can fire for instances where a student got an item right. In general, these rules 
should be written for any instance which poses a difficult problem and a student may 
need reinforcement as to how to complete the process and complete the deliverable. 

Polish 

Pohsh rules need to fire when almost everything in the target group is correct and 
the student is making small and insignificant mistakes. 

If 80%-99% of the targets in the first three journal entries are correct 
And the first three journal entries have been tried 
Then provide polish feedback 

This polish rule shows two things. First, the rule is scoped so that it will not fire 
when any of the furst three joumal entries have not been attempted. In addition, the 
rule will not fire if all of the joumal entries are 100% correct. With these boundaries 
in place the rule will only fire when the student has attempted all of the first three 
joumal entries and they are 80%-99% correct. Note: The determination of the 
exact percentages which must be correct to receive "polish" versus "focus" or 
"redirect" feedback will be determined by the designer, and are most likely 
specific to the particular task being completed. 

Focus 

Focus rules are the most common type of rule. Focus rules should fire when the 
student knows enough to complete the task but not enough to get the task correct. 

If 40%-79% of the targets in the first three joumal entries are correct 
And the first three joumal entries have been tried 
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Then provide focus feedback 

This focus rule also shows scoping. The rules are scoped to fire between 40% and 
79%. Below 40% is redirect and above 79% is polish. The rule also fires only 
5 when all of the required work has been attempted. 

Redirect 

Redirect rules should fire when it is clear that the student does not have a good 
understanding of how to complete the task. This is evidenced by a significant 
1 0 number of errors found in the student's work. 

If less than 40% of the first three journal entries are correct 
And the first three journal entries have been tried. 
Then provide redirect feedback 

15 

This redirect rule is to catch those who are truly lost. If the student has tried to 
complete all of the work, and they are less than 40% correct, then they need a great 
deal of help to continue the task. 

20 Mastermind 

Mastermind rules need to track down situations when the student is simply trying to 
cheat his way through the application. 

If less than 40% of the first three journal entries are correct 
25 And the student has made only one change twice in a row. 
Then provide mastermind feedback 



30 



This mastermind rule catches those who are making one change, asking for feedback 
over and over. One thing to keep in mind is that as a student gets towards praise 
they need to make small changes and then ask for feedback. To allow this, the 
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above rule is scoped so that if the student has more than 40% of the work right the 
rule will not fire. 

Incomplete 

5 In many activities the student should try and complete most if not all of the work 
before asking for feedback. One of the goals of many training applications is to 
mimic the real world, and it is rare for an employee to ask for a review after every 
little step they complete. Most employers want to see a significant amount of work 
done before asking for a review. 

10 

If all of joumal entries have NOT been tried, 
Then provide incomplete feedback 

Forcing a student to attempt all of his work first will require him to gain confidence 
15 in his abiUty to complete the work. Therefore, incomplete rules should be used after 
baby-step feedback so that students feel that they have the tools and abihty to 
complete the whole task before asking for feedback. 

Principles of Rule Design 
20 There are a couple of general rules which make rule creation and maintenance easier. 

Use percentages whenever possible 

It may seem easier at the time to write rules which look for specific numbers of right 
and wrong items. But when a rale is written which looks for a specific number, it 
25 means that if the data ever changes, you will need to get back into that rale and 

tweak it so that it still fires at the right time. It is far better to write percentage rales 
which fire whenever a certain percentage of work is either right or wrong. Then if 
the data ever changes and more right answers are added or some removed, then the 
rales may not need to be rewritten. 



30 
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Scope the rules as tightly as possible 

As stated previously, it is very important to make the rules mirror the written 
feedback. If the feedback is vaguer than the rule, then the students will not 
understand the exact nature of the problem. The feedback will simply hint at it If 
5 the feedback is more specific than the rule, students will become confused. The 
student may not have made the specific error the feedback is referring to under the 
umbrella rule. 

Data Dictionary In Accordance With 

A Preferred Embodiment 

1 0 Domain Knowledge Model Data Dictionary 



Column 


Type 


Len 


Description 


Source 




SourcelD 


Counter 




Unique key for this table 


Source 


String 


50 


Name of this object 


SourceDesc 


String 


255 


Documentation String that 
appears with this object in 
auto-documentation reports 


SourceCaption 


String 


50 


String that can be dynamically 
embedded into feedback text 
using Parameter Substitution 
Language (PSL) 


Sourceltem 




SourceltemID 


Counter 




Unique key for this table 


Sourceltem 


String 


50 


Name of this Object 


SourceltemDesc 


String 


255 


Documentation String that 
appears with this object in 
auto-documentation reports 


SourceltemText 


String 


50 


String that Can be dynamically 
embedded into feedback text 
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using Parameter Substitution 
Language (PSL) 


TargetPage 




TargetPagelD 


Counter 




Unique key for this table 


TargetPage 


String 


50 


Name of this object 


TargetPageDesc 


String 


255 


Documentation String that 
appears with this object in 
auto-documentation reports 


TargetPageCapt 
ion 


String 


50 


String that Can be dynamically 
embedded into feedback text 
using Parameter Substitution 
Language (PSL) 




Target 




TargetID 


Counter 




Unique key for this table 


Target 


String 


50 


Name of this object 


TargetDesc 


String 


255 


Documentation String that 
appears with this object in 
auto-documentation reports 


TargetCaption 


String 


50 


String that Can be dynamically 
embedded into feedback text 
using Parameter Substitution 
Language (PSL) 


SourceltemTar 
get 




SourceltemID 


Long 




SourceltemID of the 
association 


TargetID 


Long 




TargetID of the association 


Relevance 


Float 




Value between -1 and 1 that 
indicates the relative relevance 
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of this association between a 
Sourceltem and a Target A 
negative value indicates that 
this association is incorrect. A 
positive value indicates that it 
is correct. A value of zero 
indicates that this association is 
irrelevant. 




Attribute 




SourceltemID 


Long 




SourceltemID of the 
association 


TargetID 


Long 




TargetID of the association 


AttributelD 


Counter 




Unique key for this table 


Attribute 


String 


50 


Name of this object 


Correctind 


Bool 




Boolean value that indicates 
whether this Attribute is 
correct or incorrect for this 
association of Sourceltem and 
Target 


AttributeMin 


Double 




The lower bound for the range 
of this attribute. 


AttributeMax 


Double 




The upper bound for the range 
of this attribute. 




ControlSourcel 
tem 




ModuleName 


String 


50 


Name of module the control is 
on 


ControlName 


String 


50 


Name of Control the 
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Sourceltem is mapped to 


ItemNo 


Integer 




A single control may be 
mapped to multiple 
Sourceltems depending on how 
it is viewed. If one control is 
used on four different tabs to 
show four different values, the 
ItemNo will change as the tabs 
change, but the ControlName 
will stay the same. 


SourceltemID 


Long 




ID of Sourceltem that this 
control is mapped to 


Start 


Integer 




For controls that contain text, 
this is the start position of the 
text that the Sourceltem is 
associated with. 


End 


Integer 




For controls that contain text, 
this is the end position of the 
text that the Sourceltem is 
associated with. 


TaskID 


Long 




This is the TaskID the module 
is in 


Description 


Text 


255 


Comment Information that can 
appear in the generated 
documentation reports. 




ControlTarget 




ModuleName 


String 


50 


Name of module the control is 


ControlName 


String 


50 


Name of Control the 
Sourceltem is mapped to 
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ItemNo 


Integer 




A single control may be 
mapped to multiple Targets 
depending on how it is viewed. 
If one control is used on four 
different tabs to show four 
different values, the ItemNo 
will change as the tabs change, 
but the ControlName will stay 
the same. 


TargetID 


Long 




ID of Target that this control is 
mapped to 


Start 


Integer 




For controls that contain text, 
this is the start position of the 
text that the Target is 
associated with. 


End 


Integer 




For controls that contain text, 
this is the end position of the 
text that the Target is 
associated with. 


TaskID 


Long 




This is the TaskID the module 
is in 


Description 


Text 


255 


Comment Information that can 
appear in the generated 
documentation reports. 



Student Data Model Data Dictionary 

Column Type Len Description 



Student 




SourcelD 


Counter 




Unique key for this table 
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Source 


String 


50 


Name of this object 


SourceDesc 


String 


255 


Documentation String that 
appears with this object in 
auto-documentation reports 


SourceCaption 


String 


50 


String that Can be 
dynamically embedded into 
feedback text using Parameter 
Substitution Language (PSL) 




StudentSubmissi 
on 




SourceltemlD 


Counter 




Unique key for this table 


Sourceltem 


String 


50 


Name of this Object 


SourceltemDesc 


String 


255 


Documentation String that 
appears with this object in 
auto-documentation reports 


SourceltemText 


String 


50 


String that Can be 
dynamically embedded into 
feedback text using Parameter 
Substitution Language (PSL) 




UserSourceltemTar 
get 




SourceltemlD 


Counter 




Unique key for this table 


Sourceltem 


String 


50 


Name of this Object 


SourceltemDesc 


String 


255 


Documentation String that 
appears with this object in 
auto-documentation reports 


SourceltemText 


String 


50 


String that Can be 
dynamically embedded into 
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feedback text using Parameter 
Substitution Language (PSL) 
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Rule Model Data Dictionary 



Column 


Type 


Len 


Description 


Rule 




TaskID 


Long 




ED of Task for which this 
rule is in scope 


CoachID 


Long 




ID of Coach for which this 
rule is in scope 


RulelD 


Counter 




Unique key for this table 


Rule 


String 


50 


Name of this object 


RuleDesc 


String 


255 


Documentation String that 
appears with this object in 
auto-documentation 
reports 


RuleCondCountMin 


Integer 




Minimum number of 
conditions that must be 
true for this Rule to fire 


RuleCondCountMax 


Integer 




Maximum number of 
conditions that must be 
true for this Rule to fire 


CoachTopicID 


Long 




ID of CoachTopic that is 
activated when this rule 
fires 




RuleAggregateAnds 




RulelD 


Long 




ID of Rule of which this 
object is a condition 


RuleCondID 


Counter 




Unique key for this table 


TargetGroupID 


Long 




ID of TargetGroup whose 
aggregate values are 
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compared to the aggregate 
boundaries of this 
condition 


AggRelevanceMin 
AggRelevanceMax 


Float 




The TargetGroup's 
Calculated Aggregate 
Relevance must fall 

V>ptwpf=*"n tViiG AyfiTi anrl A/Tqv 
Ut^LWC/dl tlllo iVllll aiiU. iVidA 

for this condition to be 
true 


AggUserCntPosMin 
AggUserCntPosMax 


Integer 




The oositive-relevance 
associations the user has 


made using Targets in this 
TargetGroup are counted 
to produce an Aggregate 
value called *UserCntPos\ 
This TargetGroup's 
UserCntPos must fall 
between this condition's 
rvgguserv^nxr^osiVim ana. 
AggUserCntPosMax for 
this condition to be true. 


AggUserCntNegMin 
AggUserCntNegMax 


Integer 




The negative-relevance 
associations the user has 


made using Targets in this 
TargetGroup are counted 
to produce an Aggregate 
value called 
'UserCntNeg'. This 
TargetGroup 's 
UserCntNeg must fall 
between this condition's 
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AggUserCntNegMax for 
this condition to be true. 


AggUserCntZeroMin 
AggUserCntZeroMax 


Integer 




The zero-relevance 
associations the user has 


made using Targets in this 
TargetGroup are counted 
to produce an Aggregate 
value called 
'UserCntZero'. This 
TargetGroup 's 
UserCntZero must fall 
between this condition's 
AggUserCntZeroMin and 
AggUserCntZeroMax lor 
this condition to be true. 


AggUserSumPosMin 
AggUserSumPosMax 


Float 




The relevance values of 
the Dositive-relevance 
associations the user has 


made using Targets in this 
TargetGroup are summed 
to produce an Aggregate 
value called 
'UserSumPos'. This 
TargetGroup's 
UserSumPos must fall 
between this condition's 
AggUserSumPosMin and 
AggUserSimiPosMax for 
this condition to be true. 


AggUserSumNegMin 


"loat 




The relevance values of 
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AggUserSumNegMax 






the negative-relevance 
associations the user ha5; 
made using Targets in this 
TargetGroup are summed 
to produce an Aggregate 
value called 
'UserSumNeg'. This 
TargetGroup 's 
UserSumNeg must fall 
between this condition's 
/^ggubcrounLTNegivim ana 
AggUserSumNegMax for 
this condition to be true. 


AggUserCntPos2Min 
AggUserCntPos2Max 


Integer 




The r>ositive-relevance 
associations the user has 


made using Targets in this 
TargetGroup where the 
user's Attribute are 
coimted to produce an 
Aggregate value called 
'UserCntPos2'. This 
TargetGroup's 
UserCntPos2 must fall 
between this condition's 
AggUserCntPos2Min and 
AggUserCntPos2Max for 
this condition to be true. 


RuleSpecificMapping 
Ands 




RulelD 


Long 




ID of Rule of which this 
object is a condition 
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SourceltemID 


Long 




SourceltemID of the 
association 


TargetlD 


Long 




TargetlD of the 
association 


SourceltemID 


Long 




Unique key for this table 


AttributeMatchType 


Byte 






AttributelD 


Long 




Documentation String that 
appears with this object in 
auto-documentation 
reports 


AttributeMatchType 




AttributeMatchType 


Byte 




Unique key for this table 


AttributeMatchTypeD 
esc 


String 


255 


Brief text description of 
each AttributeMatchType 
Type 
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Feedback Model Data Dictionary 

Column Type Len Description 



CoachToDic 




TaskID 


Long 




ID of Task for which this 
object is in scope 


1 argenjrroupiJj 


Long 




ID of TargetGroup which this 
topic of remediation relates to 


CoachTopicID 


Counter 




Unique key for this table 


CoachTopic 


String 


50 


Name of this object 


CoachTopicDesc 


String 


255 


Documentation String that 
appears with this object in 
auto-documentation reports 


CoachTopicPriori 

ty 


String 


3 


Priority of this CoachTopic 
with respect to other 
CoachTopics in the same 
TargetGroup 


RemediationType 


String 


50 


Type of remediation that this 
CoachTopic is. This 
determines how the 
CoachTopic is handled at 
runtime. 


CoachltemStandA 
loneReentrySeqID 


String 


50 


When all the Stand Alone 
Coachltems in this 
CoachTopic have been used, 
they are restarted on the 
CoachltemStandAloneReentry 
SeqID. If the 

CoachltemStandAloneReentry 
SeqID = 0 the StandAlone 
half of the CoachTopic is 
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expired and no longer used. 


CoachltemChildR 
eentrySeqID 


String 


50 


When all the Child 
Coachltems in this 
CoachTopic have been used, 
they are restarted on the 
CoachltemChildReentrySeql 
D, If the 

CoachltemChildReentrySeql 
D = 0 the Child half of the 
CoachTopic is expired and no 
longer used. 




RemediationTyp 
e 




SourceltemID 


Counter 




Unique key for this table 


Sourceltem 


String 


50 


Name of this Object 


SourceltemDesc 


String 


255 


Documentation String that 
appears with this object in 
auto-documentation reports 


SourceltemText 


String 


50 


String that Can be 
dynamically embedded into 
feedback text using Parameter 
Substitution Language (PSL) 




Coachltem 




SourceltemE) 


Counter 




Unique key for this table 


Sourceltem 


String 


50 


Name of this Object 


SourceltemDesc 


String 


255 


Documentation String that 
appears with this object in 



-122- 









auto-documentation reports 


SourceltemText 


String 


50 


String that Can be 
dynamically embedded into 
feedback text using Parameter 
Substitution Language (PSL) 





Source Code In Accordance With A Preferred Embodiment 

llllllilllllllllllllllllllllllllllllllllllllllllllllllllllllllllljllllllllllllllllllllll 
II tutxport.h 



lllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllljl 

II Control Functions 

/* 

* Name: TuResumeStudent 

* Purpose: To Resume a Student In progress. 

* Author: Mike Smialek / Andersen Consulting 

* Input 

* Parameters: long StudentID 

* The Unique ID of the Student to load 
* 

* long TaskID 

* The Unique ID of the Task to Load 

* int fromSubmissionSeqID 

* The Submission from which the Student continues the Task 

* <0 :Resume Task from latest submission 

* =0 :Restart Task 

>0 :Continue from a specific submission 
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* Output 

* Parameters: none 

* Function Return 

* Variables: TUT_ERR_DB_COULDNT_OPEN_DATABASE 
5 * TUT_ERR_DOC_COULDNT_LOAD_TASK_DOC 

* TUT_ERR_LOD_NO_COACHTOPICS_FOUND 

* TUT_ERR_LOD_NO_COACHITEMS_FOUND 

* TUT_ERR_LOD_NO_COACHES_FOUND 

* TUT_ERR_LOD_NO_SOURCEITEMTARGETS_FOUND 
10 * TUT_ERR_LOD_NO_SOURCES_FOUND 

* TUT_ERR_LOD_NO_SOURCEITEMS_FOUND 

* TUT_ERR_L0D_N0_TARGETGR01IPS_F0UND 

* TUT_ERR_LOD_NO_TARGETS_FOUND 

* TUT_ERR_LOD_NO_TARGETPAGES_FOUND 

15 * TUT_ERR_LOD_NO_TARGETGROUPTARGETS_FOUND 

* TUT_ERR_LOD_NO_RULES_FOUND 

* TUT_ERR_DB COULDNT OPEN RECORDSET 



20 



TUT ERR OK 



* Notes: Loads from Database or Document based on values 

* of m_StorageTypeTask and m_StorageTypeStudent 
* 

25 */ 

extern "C" 
{ 

long _export WINAPI TuResumeStudent(long StudentID, long TaskID, int 
fromSubmissionSeqID ); // Resumes a Student's work for the Task at the specified 
30 Submission 

} 
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extem "C" 
{ 

long ^export WINAPI TuLoadArchivedSubmissions(long StudentID, long 

5 TaskID, int fromSubmissionSeqlD, int toSubmissionSeqID); // Loads Archived 
Submissions For a Student's work in a Task 
} 

extern "C" 
10 { 

long export WINAPI TuUseArchivedSubmissions(int n); // Replays n 

|3 Archived submissions for debugging 

I > 

m 15 extern "C" 

i ^ 

%^ long _export WINAPI TuSaveCurrentStudentO; // Saves Current Student's 

IB work to DB 

I } 
p 20 

extern "C" 
{ 

long _export WINAPI TuSimulateStudent(long StudentID, long TaskID, float 
Intelligence, float Tenacity, int MaxTums); // Not operational 
25 } 

extern "C" 
{ 

long _export WINAPI TuWriteUserDebuglnfoQ; // writes active CoachTopics 
30 to DB for Debugging 

} 
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extem "C" 
{ 

long _export WINAPI KillEngine( long ITasklD); // Delete all Dynamic 
objects before shutdown 
} 



/* 

* Name: LoadTasklnfo 

* Purpose: To load data for a Task only. Student data is not loaded 

* Author: Mike Smialek / Andersen Consulting 

* Input 

* Parameters: long TaskID 



* 



The Unique ID of the Task to Load 



* Output 

* Parameters: none 



* Function Return 

* Variables: TUT_ERR_DB_COULDNT_OPEN_DATABASE 
TUT_ERR_DOC_COULDNT_LOAD_TASK_DOC 
TUT_ERR_LOD_NO_COACHTOPICS_FOUND 
TUT_ERR_LOD_NO_COACHITEMS_FOUND 

* TUT_ERR_LOD_NO_COACHES_FOUND 

* TUT_ERR_LOD_NO_SOURCEITEMTARGETS_FOUND 

* TUT_ERR_LOD_NO_SOURCES_FOUND 

* TUT_ERR_LOD_NO_SOURCEITEMS_FOUND 
TUT_ERR_LOD_NO_TARGETGROUPS_FOUND 
TUT_ERR_LOD_NO_TARGETS_FOUlS[D 
TUT_ERR_LOD_NO_TARGETPAGES_FOUND 
TUT ERR LOD NO TARGETGROUPTARGETS FOUND 
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* TUT_ERR_LOD_NO_RULES_FOUND 

* TUT__ERR__DB_COULDNT_OPEN__RECORDSET 

* TUT_ERR_OK 

* Notes: 

*/ 

extern "C" 

{ 

long _export WINAPI LoadTaskInfo( long ITasklD ); // Clear and (re)load info 
for TaskID 

} 

/* 

***************************************** 
* 

* Name: TuLoadTaskDoc 

* Purpose: Loads a Tutor Document containing Task Data 

* Author: Mike Smialek / Andersen Consulting 

* Input 

* Parameters: long ITasklD 

* TaskID To Load 

* Output 

* Parameters: none 

* 

* Function Return 

* Variables: TUT_ERR_DOC_COULDNT_LOAD_TASK_DOC 

* TUT_ERR_LOD_NO_COACHTOPICS_FOUND 

* TUT_ERR_LOD_NO_COACHITEMS_FOUND 

* TUT_ERR_LOD_NO_COACHES_FOUND 

* TUT ERR LOD NO SOURCEITEMTARGETS FOUND 
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* TUT_ERR_LOD_NO_SOURCES_FOUND 

* TUT_ERR_LOD_NO_SOURCEITEMS_FOUND 

* TUT_ERR_LOD_NO_TARGETGROUPS_FOUND 
TUT_ERR_LOD_NO_TARGETS_FOUND 
TUT_ERR_LOD_NO_TARGETPAGES_FOUND 
TUT_ERR_LOD_NO_TARGETGROUPTARGETS_FOUND 
TUT_ERR_LOD_NO RULES FOUND 



* TUT_ERR_OK 
10 * 

* Notes: TaskID is used to format the file name of the Document. 



***************************************** 

*/ 

15 extern "C" 
{ 

long _export WINAPI TuLoadTaskDoc( long ITasklD ); // Clear and (re)load 
info for TaskID from TaskDoc 

} 

20 

/* 

***********************:(. :(c:|s:).:(s:(,s^^^,5|(^jj.^5,.^^^^ 

* Name: TuSaveTaskDoc 

* Purpose: Saves The Task data as a Tutor Document 
25 * Input 

* Parameters: long ITasklD 

* TaskID To Save 

* Output 

* Parameters: none 
30 * 

* Function Return 
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* Variables: TUT_ERR_DOC_COULDNT_SAVE_TASK__DOC 
* 

* TUT_ERR__OK 
* 

* Notes: TaskID is currently only used to format the file name of the 
Document. 

* If a TaskID is passed in that is different than the loaded Task, 

* it will save the loaded data as if it were data for Task ID 

extern "C" 
{ 

long _export WINAPI TuSaveTaskDoc( long ITasklD ); // Save info for 
TaskID into TaskDoc 

} 



* Name: 

* Purpose: 

* Input 

* Parameters: 

* 

* Output 

* Parameters: 



TuGo 

Kicks off Submission or Secret Submission 

long ICoachlD 
CoachID submitting to 
>0 ; Submit to Specific Coach 
=0 : Secret Submission to all Coaches 

none 



* Function Retum 

* Variables: TUT ERR OK 
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* Notes: 
*/ 

extern "C" 
5 { 

long _export WINAPI TuGo( long ICoachlD ); // kick off algorithm 

} 



10 



15 



20 



25 



* Name: 

* Purpose: 

* Input 

* Parameters: 

* 

* Output 

* Parameters: 



TuIsDirty 

Gets the Dirty Status of the Task or of an individual Coach 

long ICoachlD 
CoachID for which to determine Dirty Status 
>0 :Determines Dirty Status for specific Coach 
=0 :Determines Dirty Status for whole Task 



LPINT IsDirty 

TRUE indicates this Coach or Task is Dirty 
FALSE indicates this Coach or Task is not Dirty 
If one or more Coaches is dirty the Task is Dirty 

* Function Return 

* Variables: TUT_ERR_LOD_NO_COACHES_FOUND 

* TUT_ERR_LOD__COACHID_NOT_FOUND 

* TUT ERR OK 



* Notes: 

*/ 

30 extern "C" 
{ 
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long _export WINAPI TuIsDirty(long TaskID, long ICoachlD, LPINT IsDirty 



); 



* Name: 

* Purpose: 

* Author: 

* Input 

* Parameters: 

* Output 

* Parameters: 



TuGetSubmissionSeqID 

Returns the current SubmissionSeqID 

Mike Smialek / Andersen Consulting 

long TaskID 

The TaskID for which you want the SubmissionSeqID 
none 



* Function Return 

* Variables: SubmisisonSeqID of the current Submission 



* Notes: 

*/ 

extern "C" 

{ 

long _export WINAPI TuGetSubmissionSeqID(long TaskID); 

} 



/* 

* Name: TuGetFeedbackPrevCoachID 

* Pmpose: Returns the CoachID of The Coach That delivered the previous 
feedback 

* Function Return 
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* Variables: CoachID of The Coach That delivered the previous feedback 

* Notes: 

***************************************** 

*/ 

extern "C" 
{ 

long _export WINAPI TuGetFeedbackPrevCoachlDQ; 

} 



************* *^ ,{::(: :i:5^j^^ 5[:5[; ^ 5^jj, ^5,; 5j; ^jj. 



* Name: 

* Purpose: 

* Input 

* Parameters: 

* 

* Output 

* Parameters: 
* 

* 

* 

* 

* 

* 

* 

* 



TuGetApprovalStatus 

Gets the Approval Status of the Task or of an individual Coach 

long ICoachlD 
CoachID for which to determine Approval 
>0 :Determines approval for specific Coach 
=0 :Determines approval for whole Task 

LPINT Appro valRequired 

TRUE indicates this Coach or Task requires approval 
FALSE indicates this Coach or Task does not require approval 
(Always TRUE when input CoachID == 0) 

LPINT Approved 

TRUE indicates this Coach or Task is approved 
FALSE indicates this Coach or Task is not approved 



* Function Return 

* Variables: TUT_ERR_LOD__NO_COACHES_FOUND 

* TUT ERR LOD COACHID NOT FOUND 
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* 



TUT ERR OK 



* Notes: 

*/ 

extern "C" 
{ 

long _export WINAPI TuGetApprovalStatus( long ICoachlD, LPINT 
ApprovalRequired, LPINT Approved ); // return approval status for CoachID 
} 

/* 

* Name: TuCanProceed 

* Purpose: Determines if Task is in state in which user can proceed to 
another Task 

* Input 

* Parameters: 
* 

* Output 

* Parameters: 
* 



long ITasklD 
TaskID to examine 



LPINT CanProceed 
TRUE indicates user can proceed from this Task 

* FALSE indicates user can not proceed from this Task 

* Function Return 

* Variables: TUT__ERR__LOD_NO COACHES FOUND 



* TUT__ERR_OK 

* Notes: 

*/ 

extern "C" 
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{ 



long _export WINAPI TuCanProceed( long ITasklD, LPINT CanProceed ); 



} 

/* 



5 

* Name: TuMenu 

* Purpose: Opens Menu Dialog 

* Author: Mike Smialek / Andersen Consulting 

* Input 

10 * Parameters: none 

* Output 

* Parameters: none 

* Function Return 

* Variables: TUT_ERR__OK 
15 * Notes: 

*/ 

extern "C" 
{ 

20 long _export WINAPI TuMenuO; 

} 



/* 

25 * Name: TuTesterComment 

* Purpose: Opens Tester Comment Dialog 

* Input 

* Parameters: none 

* Output 

30 * Parameters: none 

* Fxmction Return 
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* Variables: TUT ERR OK 

* Notes: 

*/ 

5 extern "C" 
{ 

long export WINAPI TuTesterCommentO; 

} 



10 /////////////////////////////^^^^^^ 

II Notification Functions 
/* 

1 5 * Name: TuCreateMap 

* Purpose: To Create an association between a Sourceltem and a Target 

* with a modifying Attribute value 

* Input 

* Parameters: long SIID 

20 * SourceltemID of existing association to create 



30 



25 * double Attr 



long TID 

TargetID of association to create 



Attribute value of association to create 



* Output 

* Parameters: none 
* 

* Function Return 
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* Variables: TUT__ERR_TUF_USIT_TARGET_NOT_FOUND 

* TUT_ERR_TUF_USIT_DUPLICATE_FOUND 

* TUT_ERR_OK 
5 * Notes: 

***************************************** 

*/ 

extern "C" 
{ 

1 0 long _export WINAPI TuCreateMap( long SIID, long TID, double Attr ); 

} 

/* 

******************************;{::(; :i::{:ijc5j:^:ji:i:^sjj 

15 * Name: TuModifyMap 

* Purpose: To Modify an association between a Sourceltem and a Target 

* with a new modifying Attribute value 

* Input 

* Parameters: long SIID 

20 * SourceltemID of existing association to Modify 

* long TID 

* TargetID of existing association to Modify 

* double Attr 

* New Attribute value for association 
25 * Output 

* Parameters: none 

* Function Return 

* Variables: TUT_ERR_TUF_USIT_TARGET_NOT_FOUND 

* TUT_ERR_TlIF_USIT_DUPLICATE_FOUND 
30 * 

* TUT ERR OK 
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* 

* Notes: This function calls TuDeleteMap / TuCreateMap 
* 

***************************************** 
5 */ 

extern "C" 

{ 

long _export WINAPI TuModifyMap( long SIID, long TID, double Attr ); 

} 

10 

/* 

* Name: TuDeleteMap2 

* Purpose: To Delete an association between a Sourceltem and a Target 
15 * Input 

* Parameters: long SIID 

* SourceltemID of association to delete 

* long TID 

20 * TargetlD of association to delete 

* double Attr 

* Attribute value of association to delete 

* Output 

25 * Parameters: none 

* Function Return 

* Variables: TUT_ERR_TUF_USIT_TARGET_NOT_FOUND 

* TUT_ERR_TUF_USIT_NOT_FOUND 

30 * TUT_ERR_OK 
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* Notes: This function ignores the Attribute value and calls 

* TuDeleteMap( long SIID, long TID ) 
***************************************** 

*/ 

extern "C" 

{ 

long _export WINAPI TiiDeleteMap2( long SIID, long TID, double Attr ); 

} 

/* 

********************************:(: :j::j(:(:j(::jt5|::j;5jt 

* Name: TuDeleteMap 

* Purpose: To Delete and association between a Sourceltem and a Target 

* Input 

* Parameters: long SIID 

* SourceltemlD of association to delete 
* 

* long TID 

* TargetID of association to delete 
* 

* Output 

* Parameters: none 
* 

* Function Return 

* Variables: TUT_ERR_TUF_USIT_TARGET_NOT_FOUND 

* TUT_ERR_TUF_USIT_NOT_FOUND 

* TUT_ERR_OK 

* Notes: 

******** *********************sj:5jc:(::|:^:fc:tc:jist::[:5j-5[c 

*/ 

extern "C" 
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{ 

long _export WINAPI TuDeleteMap( long SIID, long TID ); 

} 

5 IIIIIIIIIIIIIIIIIIIIIIIHIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIH 
lllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllll^ 
II Configuration Functions 
/* 

10 * Name: TuSetODBCConnect 

* Purpose: To set ODBC Connect String for the Task Data Database 

* Input 

* Parameters: LPCSTR ODBCConnect 

* ODBC Connect String for the Task Data Database 
15 * Output 

* Parameters: none 

* 

* Function Return 

* Variables: TUT_ERR_OK 
20 * 

* Notes: 

*/ 

extern "C" 
25 { 

long _export WINAPI TuSetODBCConnect( LPCSTR ODBCConnect ); 

} 



30 



/* 

* Name: TuSetODBCConnectTrack 



-139- 



* Purpose: To set ODBC Connect String for the Student Tracking Database 

* Input 

* Parameters: LPCSTR ODBCConnect 

* ODBC Connect String for the Student Tracking Database 
5 * Output 

* Parameters: none 

* Fxmction Retiun 

* Variables: TUT_ERR_OK 

* Notes: 

*/ 

extern "C" 
{ 

long _export WINAPI TuSetODBCConnectTrack( LPCSTR ODBCConnect ); 

15 } 

/* 

* Name: TuSetTaskDocPathName 

20 * Purpose: To set path and name of the Task Document file 

* Input 

* Parameters: LPCSTR fhm 

* Path and name of the Task Document file 

* Output 

25 * Parameters: none 

* 

* Function Return 

* Variables: TUT_ERR_OK 

30 * Notes: 

***************************************** 
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*/ 

extern "C" 

{ 

long _export WINAPI TuSetTaskDocPathName( LPCSTR fiim ); 

} 

/* 

* Name: TuSetFeedbackFileName 

* Purpose: To set path and name of file to use for holding feedback 

* Input 

* Parameters: LPCSTR firni 

* Path and name of file to use for holding feedback 

* Output 

* Parameters: none 

* Function Return 

* Variables: TUT_ERR_OK 
* 

* Notes: 

*/ 

extern "C" 
{ 

long _export WINAPI TuSetFeedbackFileName( LPCSTR fiim ); 

} 



/* 

* Name: TuSetFeedbackPrevFileName 

* Purpose: To set path and name of file to use for holding previous feedback 

* Input 
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* Parameters: LPCSTR fim 

* Path and name of file to use for holding previous feedback 

* Output 

* Parameters: none 
5 * 

* Function Return 

* Variables: TUT_ERR_OK 

* Notes: 

^ ^ ^fc SjC 3^ «^ s{« «{* Sfc sfc 3^ ^[t S]>C «{c S}t Sfc S{S Sfc sfc tff* 

10 */ 

extern "C" 
{ 

long _export WINAPI TuSetFeedbackPrevFileName( LPCSTR fiun ); 

/* 

* Name: TuSetLogFileName 

* Purpose: To set path and name of file to use for full logging 

* Input 

* Parameters: LPCSTR fiim 

20 * Path and name of file to use for fixU logging 

* Output 

* Parameters: none 

* 

* Function Return 

25 * Variables: TUT_ERR_OK 

* Notes: 

*/ 

extern "C" 
30 { 

long _export WINAPI TuSetLogFileName( LPCSTR fiim ); 
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* Name: TuSetLogLoadFileName 

5 * Purpose: To set path and name of file to use for load logging 

* Input 

* Parameters: LPCSTR fimi 

* Path and name of file to use for load logging 

* Output 

10 * Parameters: none 

* Function Retum 

* Variables: TUT_ERR__OK 

15 * Notes: 
*/ 

extem "C" 
{ 

20 long _exportWINAPITuSetLogLoadFileName( LPCSTR film); 
} 

/* 

25 * Name: TuSetLogStudentFileName 

* Purpose: To set path and name of file to use for student logging 

* Input 

* Parameters: LPCSTR fiim 

* Path and name of file to use for student logging 
30 * Output 

* Parameters: none 
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* Function Return 

* Variables: TUT_ERR_OK 

* 

5 * Notes: 

***************************************** 

*/ 

extern "C" 

{ 

10 long _export WINAPI TuSetLogStudentFileName( LPCSTR fhm ); 

} 

/* 

15 *Name: TuSetLogSubmissionFileName 

* Purpose: To set path and name of file to use for submission logging 

* Input 

* Parameters: LPCSTR fhm 

* Path and name of file to use for submission logging 
20 * Output 

* Parameters: none 
* 

* Function Return 

* Variables: TUT„ERR_OK 
25 * 

* Notes: 

***************************************** 

*/ 

extern "C" 
30 { 

long _export WINAPI TuSetLogSubmissionFileName( LPCSTR fiun ); 
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»jj *1* mS^ «ip «L* ^ *£# ^ «Xr «1> •X' *X» ^ ^ «^ ^ *lf ^ ^ ^tt Sle* ^ *1> ^ ^ )l> <U <t ^ 
?J> *T* *p *T' *T* *T* *T* *T* ^ *T* *T* •T' *T* *¥* 'I* 'T* *I* 'T* "T* *T* *** *(* *T* *T* *r •** *T* *T* *r* 1* *T* 

5 * Name: TuSetLogErrFileName 



* Purpose: 

* Input 

* Parameters: 

10 * Output 

* Parameters: 



To set path and name of file to use for error logging 

LPCSTR fhm 
Path and name of file to use for error logging 

none 



* Function Return 

* Variables: TUT_ERR_OK 
15 * Notes: 

^i*- ^Im -kl* *±f ^ ^ «£f -i* *i* ^ slf ^> *i- •!> ^ *i> *||^ ^ ^ ^ ^ ^ 

?I> vj^ ^> *|> 5|> *|> ^f- *tS *T* *T* *f* *T* *T* *T* "T* *t* "T" "T" *T" *T* *•* *** *T* •V' *T* •T* *T' *T" 

*/ 

extern "C" 
{ 

20 long _export WINAPI TuSetLogErrFileName( LPCSTR fhm ); 
} 



30 



si:**:!:* Hi** ************************** ******* 



25 * Name: 

* Purpose: 

* Input 

* Parameters: 
* 
* 



Output 



TuSetTrace 

To tum Trace on and off 

int TraceStatus 
TUT_TRACE_ON :Tum Trace On 
TUT TRACE OFF :Tum Trace Off 
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* Parameters: 



none 



* Function Return 

* Variables: Previous Trace Status Value 

* TUT_TRACE_ON 

* TUT TRACE OFF 



* TUT_ERR_INVALID_TRACE_STATUS 

* Notes: 

J^Q ***************************************** 

*/ 

extern "C" 
{ 

long _export WINAPI TuSetTrace( int TraceStatus ); 

15 } 

/* 

***************************************** 



20 



25 



30 



* Name: 

* Purpose: 
* 

* 

* Input 



TuSetTrack 

To turn Tracking on and off. WWle tracking is on 
all work the user does and all feedback the user receives 
is kept. While Tracking is off only the most recent work is kept. 



* Parameters: int TrackStatus 

* TUT„TRACK„ON :Tum Tracking On 

* TUT_TRACK_OFF :Tum Tracking Off 

* Output 

* Parameters: none 

* Function Return 

* Variables: Previous Trace Status Value 

* TUT_TRACK_ON 

* TUT TRACK OFF 
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* TUT_ERR_INVALID_TRACK__STATUS 

* Notes: 

H:**************************s*t** *********** 

5 */ 

extern "C" 
{ 

long _export WINAPI TuSetTrack( int TrackStatus ); 

} 

10 

/* 

jfc^^H? ************************************* 

* Name: TuSetShowExceptionPopup 

* Purpose: To Exception popups on and off. 
15 * Input 

* Parameters: int PopupStatus 

* TUT_POPUP_ON :Tum Exception popups On 

* TUT_POPUP_OFF :Tum Exception popups Off 

* Output 

20 * Parameters: none 
* 

* Function Return 

* Variables: Previous Exception popup Status Value 

* TUT__POPlIP_ON 

25 * TUT_POPUP__OFF 
* 

* TUT_ERR_INVALID_POPUP„STATUS 

* Notes: 

***************************************** 

30 */ 

extern "C" 
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{ 

long export WINAPI TuSetShowExceptionPopup( int PopupStatus ); 

} 



5 /* 

* Name: TuSetStorageType 

* Purpose: To Direct Task and Student data to be loaded and saved 

* using a Docunaent or Database 
10 * Input 

* Parameters: long StorageTypeTask 

* TUT_STORAGE__TYPE__DOCUMENT :Load and Save Task 
Data using Document 

* TUT_STORAGE_TYPE_DATABASE :Load and Save Task Data 
15 using Database 

* long StorageTypeStudent 

* TlJT^STORAGE_TYPE__DOCUMENT :Load and Save Student 
Data using Document 

20 * TUT_STORAGE_TYPE_DATABASE :Load and Save Student 

Data using Database 

* Output 

* Parameters: none 

25 * Function Retum 

* Variables: TUT_ERR_INVALID^STORAGE_TYPE_TASK 

* TUT_ERR_INVALID_STORAGE__TYPE_,STUDENT 

* TUT_ERR_OK 
30 * Notes: 
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*/ 

extern "C" 
{ 

long _export WINAPI TuSetStorageType( int StorageTypeTask, int 
5 StorageTypeStudent ); 

} 

lllllllllllllllllllllllllllllllllllllllllllllllllllllllll^ 

llllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllll 



10 Simulation Engine 

The idea is for the designer to model the task that he wants a student to accomplish 
using an Excel spreadsheet. Then, have an algorithm or engine that reads all the 
significant cells of the spreadsheet and notifies the IntelUgent Coaching Agent with 
the appropriate information (SourceltemID, TargetED and Attribute). This way, the 
15 spreadsheet acts as a central repository for student data, contains most of the 

calculations required for the task and in conjunction with the engine handles all the 
communication with the ICA, The task is self contained in the spreadsheet, 
therefore the designers no longer need a graphical user interface to functionally test 
their designs (smart spreadsheet). Figure 40 is a block diagram illustrating how the 
20 simulation engine is architected into a preferred embodiment of the invention. 

Once the model and feedback for it are completely tested by designers, developers 
can incorporate the spreadsheet in a graphical user interface, e.g., Visual Basic as a 
development platform. The simulation spreadsheet is usually invisible and 

25 populated using functions provided by the engine. It is very important that all 

modifications that the ICA needs to know about go through the engine because only 
the engine knows how to call the ICA. This significantly reduced the skill level 
required from programmers, and greatly reduced the time required to program each 
task. In addition, the end-product was less prone to bugs, because the tutor 

30 management was centralized. If there was a tutor problem, we only had to check on 
section of code. Finally, since the simulation engine loaded the data from a 
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spreadsheet, the chance of data inconsistency between the tutor and the application 
was nil, 

5 Object Model 

Figure 41 is a block diagram setting forth the architecture of a simulation model in 
accordance with a preferred embodiment. The Simulation Object Model consists of 
a spreadsheet model, a spreadsheet control object, a simulation engine object, a 
simulation database, input objects, output objects, list objects and path objects. 

10 

Spreadsheet object 

The first object in our discussion is the Spreadsheet object. The Spreadsheet is the 
support for all simulation models. A control object that is readily integrated with 
the Visual Basic development plat. The control supports printing and is compatible 
15 with Microsoft Excel spreadsheets. With that in mind, designers can use the power 
of Excel formulas to build the simulation. The different cells contained in the 
spreadsheet model can be configured as Inputs, Outputs or Lists and belong to a 
simulation Path. 
Input object 

20 All cells in the spreadsheet that need to be manually entered by the designer or the 
student via the GBS appUcation are represented by input objects. Every input has 
the following interface: 



Field Name 


Data Type 


Description 


InputID 


long 


Primary Key for the table 


TaskID 


long 


TaskID of the task associated with the 
input 


PathID 


long 


PathID of the path associated with the 
input 


InputName 


string*50 


Name of the input 


InputDesc 


string*255 


Description of the input 


ReferenceName 


string*50 


Name of the spreadsheet cell associated 
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with the input 


TutorAware 


boolean 


Whether the ICA should be notified of 
any changes to the input 


SourceltemID 


long 


SourceltemID if input is a distinct input 
0 if input is a drag drop input 


TargetID 


long 


TargetID of the input 


Row 


long 


Spreadsheet row number of the input 
speed optimization 


Colimui 


long 


Spreadsheet column number of the input 
speed optimization 


SheetName 


string*50 


Sheet name were the input is located 
speed optimization 



This information is stored for every input in the Input table of the simulation 
database (ICASim.mdb). Refer to the example below. 



When designers construct their simulation model, they must be aware of the fact that 

there are 2 types of Inputs: 

L Distinct Input 

2. Drag & Drop Input 

Distinct Input 

The Distinct Input consists of a single spreadsheet cell that can be filled by the 
designer at design time or by the GBS application at run time via the simulation 
engine object's methods. The purpose of the cell is to provide an entry point to the 
simulation model. This entry point can be for example an answer to a question or a 
parameter to an equation. If the cell is TutorAware (all inputs are usually 
TutorAware), the ICA will be notified of any changes to the cell. When the ICA is 
notified of a change two messages are in fact sent to the ICA: 
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1. An ICANotifyDestroy message with the input information i.e., SourceltemID, 
TargetID and null as Attribute. This message is to advise the ICA to remove this 
information from its memory. 

2. An ICANotifyCreate message with the input information i.e., SourceltemID, 
TargetID, Attribute (cell numeric value) . This message is to advise the ICA to add 
this information to its memory. 

A Distinct Input never requires that a user answer a mathematics question. These 
are the steps required to configure that simulation. Figure 42 illustrates the 
arithmetic steps in accordance with a preferred embodiment, 

1. Define a name for cell C2 in Excel. Here we have defined "Distinct_Input", 

2. In the ICA, define a task that will be assigned to the simulation. Ex: a TaskBD of 
123 is generated by the ICA. 

3. In the ICA, define a Target for the input. Ex: a TargetID of 4001 is generated by 
the ICA. 

4. In the ICA, define a Sourceltem for the input. Ex: a SourceltemED of 1201 is 
generated by the ICA. 

5. Associate the input to a path (refer to Path object discussion). 

6. Add the information in the Input table of the simulation engine database. 



A record in an Input table is presented below. 



InputID: 


12345 


TaskID: 


123 


PathID: 


1234 


InputName: 


Question 1 input 


InputDesc: 


Distinct input for Question 1 


ReferenceName: 


Distinct_Input 


TutorAware: 


True 


SourceltemlD 


1201 


TargetID: 


4001 
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Row: 


2 


Column: 


3 


SheetName: 


Sheetl 



The Row, Column and SheetName are filled in once the user clicks "Run 
Inputs/Outputs". The simulation engine decodes the defined name (Reference 
Name) that the designer entered, and populates the table accordingly. This is an 
5 important step. We had several occasions when a designer would change the layout 
of a spreadsheet, i.e., move a defined name location, then forget to perform this step. 
As such, bizarre data was being passed to the tutor; whatever data happened to reside 
in the old row and column. Once the configuration is completed, the designer can 
now utilize the ICA Utilities to test the simulation. 

10 

Drag & Drop Input 

The drag & drop input consist of two consecutive spreadsheet cells. Both of them 
have to be filled by the designer at design time or by the GBS appUcation at run time 
via the simulation engine object's methods. This type of input is used usually when 
15 the user must choose one answer among a selection of possible answers. Drag & 
drop inputs are always TutorAware. The left most cell contains the SourceltemID of 
the answer picked by the user (every possible answer needs a SourceltemID) and the 
rightmost cell can contain a numeric value associated to that answer. You need to 
define a name or ReferenceName in the spreadsheet for the rightmost cell. 

20 

ICA will be notified of any changes to either one of the cells. When the ICA is 
notified of a change two messages are in fact sent to the ICA: 

1, An ICANotifyDestroy message with the input information i.e., SourceltemID 
before the change occurred, TargetED of the input and the Attribute value before the 

25 change occurred. 

2. An ICANotifyCreate message with the input information i.e., SourceltemID after 
the change occurred, TargetID of the input and the Attribute value after the change 
occurred. 
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Let's demonstrate the use of a drag & drop input building on top of the previous 
example. Here, the user is asked to answer yet another mathematics question. These 
are the steps required to configure that section of the simulation. Figure 43 
5 illustrates a drag & drop input operation in accordance with a preferred embodiment 

1. Define a name for cell CI 1 in Excel. Here we have defined "DragDrop_Input". 

2. Let's use the same TaskID as before since Question 2 is part of the same 
simulation as Question L Ex: TaskID is 123. 

3. In the IC A, define a Target for the input Ex: a TargetID of 4002 is generated by 
10 thelCA. 

4. In the ICA, define a Sourceltem for every possible answer to the question. Ex: 
SourceltemlDs 1202 to 1205 are generated by the ICA. 

5. Associate the input to a path (refer to Path object discussion). 

6. Add the information in the Input table of the simulation engine database. 

15 

A record in the Input table in accordance with a preferred embodiment is presented 
below. 



InputID: 


12346 


TaskID: 


123 


PathID: 


1234 


InputName: 


Question 2 input 


InputDesc: 


Drag & Drop input for Question 2 


ReferenceName: 


DragDrop_Input 


TutorAware: 


True 


SourceltemID 


Q *** 


TargetID: 


4002 


Row: 


11 


Column: 


3 


SheetName: 


Sheetl 
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List object 

Figure 44 illustrates list object processing in accordance with a preferred 
embodiment. The list object consists of one cell identifying the list (cell #1) and a 
5 series of placeholder rows resembling drag & drop inputs (cells #1 . 1 - 1 .n to cells 
#n. 1- n.n). The Ust is used usually when the user must choose multiple elements 
among a selection of possible answers. Cell #1 must have a uniquely defined name 
also called the list name. Cells #1.1 to #n.l can contain the SourceltemID of one 
possible answer picked by the user (every possible answer needs a SourceltemID). 

10 The content of these cells must follow this format : -ListName-SourceltemlD. 
Cells #1.2 to #n.2 will hold the numeric value (attribute) associated with the 
SourceltemID in the cell immediately to the left. Cells #1.3 - l.n to #n.3 - n.n are 
optional placeholders for data associated with the answer. KEY NOTE: When 
implementing a list object the designer must leave all the cells under #n.l to #n,n 

15 blank because this range will shift up every time an item is removed from the hst. 



Every hst has the following interface: 



Field Name 


Data Type 


Description 


ListID 


long 


Primary Key for the table 


TaskID 


long 


TaskID of the task associated with the 
list 


PathID 


long 


PathID of the path associated with the 
Ust 


ListName 


string* 50 


Name of the list 


ListDesc 


string*255 


Description of the list 


ReferenceName 


string* 50 


Name of the spreadsheet cell associated 
with the list 


TutorAware 


boolean 


Whether the ICA should be notified of 
any changes to the list 


TargetlD 


long 


TargetlD of the output 


TotalColumns 


long 


Total number of data columns 
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Row 


long 


Spreadsheet row number of the output 
-> speed optimization 


Column 


long 


Spreadsheet column number of the 
output speed optimization 


SheetName 


string*50 


Sheet name were the input is located -> 
speed optimization 



Use of a Ust is demonstrated by continuing our math test. The math question in this 
example invites the user to select multiple elements to construct the answer. These 
are the steps required to configure that section of the simulation. Figure 45 
illustrates the steps for configuring a simulation in accordance with a preferred 
embodiment. 

1 . Define a name for cell C23 in Excel Here we have defined "The_List", 

2. Let's use the same TaskID as before since Question 3 is part of the same 
simulation as Question 1 and 2. Ex: TaskID is 123. 

3. In the ICA, define a Target for the list. Ex: a TargetID of 4006 is generated 
by the ICA. 

4. In the ICA, define a Sourceltem for every item that could be placed in the 
list Ex: the following SourceltemlDs 1209, 1210, 1211, 1212, 1213, 1214 
are generated by the ICA. 

5. Associate the list to a path (refer to Path object discussion). 

6. Add the information in the List table of the simulation engine database. 



A record in the List table in accordance with a 
preferred embodiment is presented in the table 
appearing below. 



ListID: 


12346 


TaskID: 


123 


PathID: 


1234 


ListName: 


Question 3 list 


ListDesc: 


List for Question 3 


ReferenceName: 


The_List 
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Tutor Aware: 


True 


TargetID: 


4006 


TotalColuiims: 


1 


Row: 


23 


Column: 


3 


SheetName: 


Sheet! 



Output object 

All cells in the spreadsheet that are result of calculations (do not require any external 
input) can be represented by output objects. Every output has an interface as 
outlined in the table below. 



Field Name 


Data Type 


Description 


OutputID 


long 


Primary Key for the table 


TaskID 


long 


TaskID of the task associated with the output 


PathID 


long 


PathID of the path associated with the output 


OutputName 


string*50 


Name of the output 


OutputDesc 


string*255 


Description of the output 


ReferenceName 


string* 50 


Name of the spreadsheet cell associated with the 
output 


TutorAware 


boolean 


Whether the ICA should be notified of any changes to 
the output 


SourceltemID 


long 


SourceltemID of the output 


TargetID 


long 


TargetID of the output 


Row 


long 


Spreadsheet row number of the output -> speed 
optimization 


Column 


long 


Spreadsheet column number of the output -> speed 
optimization 


SheetName 


string*50 


Sheet name were the input is located -> speed 
optimization 
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AU this information is stored for every output in the Output table of the simulation 
database (ICASim.mdb). When designers construct their simulation model, they 
must be aware of the fact that there is only 1 type of Outputs : the Distinct Output. 
Distinct Output 

A Distinct Output consists of one and only one spreadsheet cell that contains a 
formula or a result of calculations. The existence of Output cells is the main reason 
to have a simulation model. If the cell is TutorAware, the ICA will be notified of 
any changes to the cell when all outputs are processed otherwise the ICA will be 
unaware of any changes. When the ICA is notified of a change two messages are in 
fact sent to the ICA: 

1. An ICANotifyDestroy message with the output information i.e., SourceltemID, 
TargetID and null as Attribute. This message is to advise the ICA to remove this 
information from its memory, 

2. An ICANotifyCreate message with the output information i.e., SourceltemID, 
TargetID, Attribute (cell numeric value) . This message is to advise the ICA to add 
this information to its memory. As opposed to Distinct Inputs and Drag & Drop 
Inputs which notify the ICA on every change. Distinct Outputs are processed in 
batch just before asking the ICA for feedback . 

To notify the ICA of the total dollar amount of the items in the list. We definitely 
need a Distinct Output for that. The output will contain a sum formula. Figure 46 
illustrates a distinct output in accordance with a preferred embodiment. The steps 
required to configure that section of the simulation taking in consideration that the 
list is already configured are presented below. 

1 . Define a name for cell C24 in Excel. Here we have defined "Distinct_Outpuf' . 

2. Let's use the same TaskID as before since Question 3 is part of the same 
simulation as Question 1 and 2. Ex: TaskID is 123. 

3. In the ICA, define a Target for the output. Ex: a TargetID of 4005 is generated 
by the ICA. 
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4 In the IC A, define a Sourceltem for the output Ex: a SourceltemID of 1215 is 
generated by the ICA. 

5. Associate the output to a path (refer to Path object discussion). 

6. Add the information in the Output table of the simulation engine database. 

5 A record in an Output table in accordance with a preferred embodiment is presented 
below. 





OutputID: 


12347 




TaskID: 


123 




PathED: 


1234 




OutputName: 


Question 3 output 




OutputDesc: 


Distinct Output for Question 3 




ReferenceName: 


DistinctOutput 


; SB: 


Tutor Aware: 


True 


¥^ 


SourceltemID 


1215 




TargetID: 


4005 




Row: 


24 




Column: 


6 


m 


SheetName: 


Sheetl 



Path object 

Paths are used to divide a simulation model into sub-Simulations meaning that you 
10 can group certain inputs, outputs and Usts together to form a coherent subset or path. 
Every path has the following interface: 



Field Name 


Data Type 


Description 


PathID 


long 


Primary Key for the table 


TaskID 


long 


TaskID of the task associated with the path 


PathNo 


long 


Nmneric value associated to a path 


PathName 


string*50 


Name of the path 


PathDesc 


string*255 


Description of the path 
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AU this information is stored for every path in the path table of the simulation 
database (ICASim.mdb). 

Simulation Engine 

The simulation engine is the interface between the model, the simulation database 
and the InteUigent Coaching Agent. The simulation engine is of interest to the 
designer so that he can understand the mechanics of it all. But it is the developer of 
applications using the engine that should know the details of the interface (methods 
& properties) exposed by the engine and the associated algorithms. 

Once the designer has constructed the simulation model (Excel Spreadsheet) and 
configured all the inputs, outputs & hsts, he is ready to test using the test bench 
included in the ICA Utilities (refer to ICA Utilities documentation). The developer, 
in turn, needs to implement the calls to the simulation engine in the GBS application 
he's building. The following list identifies the files that need to be included in the 
Visual Basic project to use the simulation workbench : 



wSimEng.cls 


Simulation Engine class 


wSimEng.bas 


Simulation Engine module (this module was introduced 
only for speed purposes because all the code should 
theoretically be encapsulated in the class) 


wConst.bas 


Intelligent Coaching Agent constant declaration 


wDeclare.bas 


Intelligent Coaching Agent DLL interface 


wlca.cls 


Intelligent Coaching Agent class 


wica.bas 


InteUigent Coaching Agent module (this module was 
introduced only for speed purposes because all the code 
should theoretically be encapsulated in the class) 



To have a working simulation, a developer places code in different strategic areas or 
stages of the application. There's the Initial stage that occurs when the form 
containing the simulation front-end loads. This is when the simulation model is 
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initialized. There's the Modification stages that take place when the user makes 
changes to the front-end that impacts the simulation model. This is when the ICA is 
notified of what's happening. There's the Feedback stage when the user requests 
information on the work done so far. This is when the simulation notifies the ICA 
5 of all output changes. Finally, there's the Final stage when the simulation front-end 
unloads. This is when the simulation is saved to disk. 

The different stages of creating a simulation, including the Visual Basic code 
involved, are presented below. 

10 

Initial stage 

1. Creating the ICA & the simulation engine objects 

Code: 

Set moSimEngine = New classSimEngine 
15 Set moICA = New classICA 

Description: The first step in using the simulation engine is to create an instance of 
the class classSimEngine and also an instance of the class classICA. Note that the 
engine and ICA should be module level object "mo" variables. 

20 

2, Loading the simulation 

Code: 

IRet = moSimEngine.OpenSimulation(App.Path & DIR_DATA & 
FILE_SIMULATION, Me.bookSimulation) 

25 

IRet = moSimEngine.LoadSimulation(mlICATaskID, App.Path & 
DIR_DATABASE & DB_SIMULATION, 1) 

Description: After the object creation, the OpenSimulation and LoadSimulation 
30 methods of the simulation engine object must be called. The OpenSimulation 

method reads the specified Excel 5.0 spreadsheet file into a spreadsheet control. The 
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LoadSimulation method opens the simulation database and loads into memory a list 
of paths, a list of inputs, a list of outputs and a list of lists for the specific task. 
Every method of the simulation engine will return 0 if it completes successfully 
otherwise an appropriate error number is returned. 

3. Initializing and loading the Intelligent Coaching Agent 

Code: 

IRet = moICA.Initiahze(App.Path & "\" «& App.EXEName & ".ini", App.Path & 
DIR_DATABASE, App.Path & DIR ICADOC, App.Path & "\") 

IRet = moICA.LoadTask(mlICATaskID, ICAStudentStartNew) 

Description: The simulation engine only works in conjunction with the ICA. The 
Initialize method of the ICA object reads the application .ini file and sets the 
Tutor32.dll appropriately. The LoadTask method tells the ICA (Tutor32.dll) to load 
the .tut document associated to a specific task in memory. From that point on, the 
ICA can receive notifications. 

Note: The .tut document contains all the element and feedback structure of a task. 
Ex: SourcePages, Sourceltems, TargetPages, Targets, etc... 

4. Restoring the simulation 

Code: 

«Code to reset the simulation when starting over» 

«Code to load the controls on the simulation fi:ont-end» 

IRet = moSimEnguie.RunInputs(sPaths, True) 

IRet = moSimEngine.RunOutputs(sPaths, True) 

IRet = moSimEngine.RunLists(sPaths, True) 

Call moICA.Submit(0) 

Call moICA.SetDirtyFlag(0, False) 
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Description: Restoring the simulation involves many things: 

• clearing all the inputs and lists when the user is starting over 

• loading the interface with data from the simulation model 

• invoking the Runlnputs, RunOutputs and RunLists methods of the simulation 
5 engine object in order to bring the ICA to it's original state 

• calling the Submit method of the ICA object with zero as argument to trigger all 
the rules 

• calling the SetDirtyFlag of the ICA object with 0 and false as arguments in order 
to reset the user's session. 

10 

Running inputs involves going through the Hst of TutorAware inputs and notifying 
the ICA of the SourceltemID, TargetID and Attribute value of every input. 
Running hsts involves going through the hst of TutorAware Usts and notifying the 
ICA of the SourceltemlD, TargetID and Attribute value of every item in every list. 
1 5 The TargetID is unique for every item in a list. 

Running outputs involves going through the list of TutorAware outputs and 
notifying the ICA of the SourceltemID, TargetID and Attribute value of every 
output. 

20 Modification stage 

1. Reading inputs & outputs 

Code: 

Dim sDataArray(2) as string 
Dim vAttribute as variant 
25 Dim ISourceltemlD as long 
Dim ITargetLD as long 

IRet = moSimEngine.ReadReference("Distinct_Input", vAttribute, ISourceltemlD, 
ITargetlD, sDataArray) 

30 
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Description: The ReadReference method of the simulation object will return the 
attribute value of the input or output referenced by name and optionally retrieve the 
SourceltemID, TargetID and related data. In the current example, the attribute 
value, the SourceltemID, the TargetID and 3 data cells will be retrieved for the input 
5 named Distinctjnput. 

2. Modifying distinct inputs 

Code: 

Dim vAttribute as variant 
10 Dim ISourceltemlD as long 
Dim sDataArray(2) as string 

vAttribute=9999 
sDataArray(0)="Data Cell #1" 
15 sDataArray(l)="Data Cell #2" 
sDataArray(2)-"Data Cell #3" 

IRet = moSimEngine.WriteReference("Distinct_Input", vAttribute, , sDataArray) 

Description: Modifying a distinct input is as simple as callmg the WriteReference 
20 method of the simulation object passing the input name, the new attribute value and 
optionally a data array. The simulation engine takes care of notifying the ICA of the 
change, 

3, Modifying drag&drop inputs 

25 Code: 

Dim vAttribute as variant 
Dim ISourceltemlD as long 
Dim sDataArray(2) as string 

30 lSourceItemID=1202 
vAttribute=9999 
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sDataArray(0)-"Data Cell #1" 
sDataArray(l)="Data Cell #2" 
sDataArray(2)="Data Cell #3" 

IRet = moSimEngine.WriteReference("DragDrop_Input'\ vAttribute, 
5 ISourceltemlD, sDataArray) 

Description: Modifying a drag&drop input is as simple as calling the 
WriteReference method of the simulation object passing the input name, the new 
attribute value, the new SourceltemID and optionally a data array. The simulation 
1 0 engine takes care of notifying the IC A of the change. 

O 4. Reading lists 

'S Code: 

IRet = moSimEngine.ListRead(sListName, IListlndex, vAttribute, ISourceltemlD, 
11 15 ITargetlD, sDataArray) 

\f% 

:L Description: All list in the simulation model can be read one item at a time using the 

11 ListRead method of the simulation engine object. Passing the hst name and the 

ni index of the item to retrieve, the function will retum the Attribute value and 

20 optionally the SourceltemID, TargetE) and data array of the item specified. Use a 
looping structure to read entire lists into memory, or to search for and retrieve a 
particular line item. This will be done quite often as designers generally allow users 
to manipulate items fi-om Hsts. For example, if a user begins to drag an element of a 
list, you will need to retrieve this data from the list item they are dragging, 

25 

5. Modifying lists 

Code: 

IRet = moSimEngine.ListAdd(sListName, vAttribute, ISourceltemlD, sDataArray) 
IRet = moSimEngine.ListCount(sListName, ITotalltems) 
30 IRet = moSimEngine.ListModify(sListName, IListlndex, vAttribute, ISourceltemE), 
sDataArray) 
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IRet = moSimEngine.ListDelete(sListName, IListlndex) 

Description: The simulation engine object provides basic functionality to 
manipulate lists. 

The ListAdd method appends an item(SourceItemID, Attribute, Data array) to the 
list. Let's explain the algorithm. First, we find the top of the list using the Ust name. 
Then, we seek the first blank cell underneath the top cell. Once the destination is 
determine, the data is written to the appropriate cells and the ICA is notified of the 
change. 

The ListCount method returns the number of items in the specified list. The 
algorithm works exactly like the ListAdd method but returns the total number of 
items instead of inserting another element. 

The ListModify method replaces the specified item with the provided data. Let's 
explain the algorithm. First, we find the top of the list using the list name. Second, 
we calculate the row offset based on the item number specified. Then, the ICA is 
notified of the removal of the existing item. Finally, the data related to the new item 
is written to the appropriate cells and the ICA is notified of the change. 

The ListDelete method removes the specified item. The algorithm works exactly 
like the ListModify method but no new data is added and the cells (width of the list 
set by 'Total Columns') are deleted with the 'move cells up' parameter set to true. 
Keep this in mind, as designers often enter the wrong number of columns in the 
Total Columns parameter. When they overestimate the Total Columns, ListDelete 
will modify portions of the neighboring hst, which leads to erratic behavior when 
that list is displayed. 

Feedback stage 

\. Running the simulation 

Code: 
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IRet = moSiinEngine.RunOutputs(sPaths, True) 



Description: Inputs and lists notify the ICA when changes happen but not outputs. 
Therefore, the RunOutputs method must be invoked before submitting the work for 
5 feedback. 



2, Triggering the ICA Rule engine 

Code: 

lRet= moICA.Submit(lCoachID) 

10 

Description: Once the simulation has been processed, the Submit method of the ICA 
object must be called to trigger all the rules and deliver the feedback. This feedback 

^fl will be written by the Tutor32.dll to two RTF formatted files. One file for previous 

m feedback and one file for the current feedback. 

m 15 

^fl 3. Displaying ICA feedback 

;U Code: 

W Set oViewer = New CFeedbackViewer 

m o Viewer. CoachID = vlCoachlD 

1^ 20 Call oViewer.DisplayFeedBack(moApp) 

Description: The only thing required to display feedback information is to have an 
RTF control on a form and read-in the feedback files produced by the Submit 
method of the ICA object. 



25 Final stage 

1. Saving the simulation 

Code: 

IRet = moSimEngine.SaveSimulation(App.Path & DIR_DATA & 
FILE_SIMULATION) 



30 
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Description: The SaveSimulation method of the simulation engine object will save 
the specified Excel spreadsheet to disk. 

SYSTEM DYNAMICS IN ACCORDANCE WITH A PREFERRED 
EMBODIMENT 

To use system dynamics models in the architecture, an engine had to be created that 
would translate student work into parameters for these models. A complex system 
dynamics model to interact with an existing simulation architecture is discussed 
below. The system dynamics model provides the following capabiUties. 

1 . Allow designers to build and test their system dynamics models and IC A 
feedback before the real interface is built. 

2. Reduce the programming complexity of the activities. 

3. Centralize the interactions with the system dynamics models. 

System Dynamics Engine 

As with the simulation engine, the designer models the task that he/she wants a 
student to accomplish using a Microsoft Excel spreadsheet. Here, however, the 
designer also creates a system dynamics model (described later). The system 
dynamics engine will read all of the significant cells within the simulation model 
(Excel) and pass these values to the system dynamics model and the ICA. After the 
system dynamics model runs the information, the output values are read by the 
engine and then passed to the simulation model and the ICA, 

Figure 47 is a block diagram presenting the detailed architecture of a system 
dynamics model in accordance with a preferred embodiment. Once the simulation 
model, system dynamics model and feedback are completely tested by designers, 
developers can incorporate the spreadsheet in a graphical user interface, e.g., Visual 
Basic as a development platform. Figure 47 illustrates that when a student 
completes an activity, the values are passed to the system dynamics engine where 
the values are then passed to the system dynamics model (as an input), written to the 
simulation model and submitted to the ICA. When the system dynamics model is 
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played, the outputs are pulled by the engine and then passed to the simulation model 
and the ICA. Note that the simulation model can analyze the output from the system 
dynamics model and pass the results of this analysis to the ICA as well. The 
simulation model can then be read for the output values and used to update on- 
screen activity controls (such as graphs or reports). 

It is very important that all modifications that the ICA and system dynamics model 
need to knov/ about go through the engine because only the engine knows how to 
call these objects. This significantly reduces the skill level required from 
programmers, and greatly reduces the time required to program each task. In 
addition, the end-product is less prone to bugs, because the model and tutor 
management will be cenfralized. If there is a problem, only one section of code 
needs to be checked. Finally, since the engine loads the data from the spreadsheet, 
the chance of data inconsistency between the ICA, the system dynamics model and 
the application is insignificant. 

Figure 48 is graphical representation of the object model which is utilized to 
instantiate the system dynamic engine in accordance with a preferred embodiment. 
The System Dynamics Object Model consists of a spreadsheet object, a system 
dynamics object, a simulation database, model objects, parameter input objects and 
parameter output objects. The first object in our discussion is the Spreadsheet object. 
The Spreadsheet is the support for all simulation models. The spreadsheet object is 
integrated with a Visual Basic development platform in accordance with a preferred 
embodiment. The control supports printing and is compatible with Microsoft Excel 
spreadsheets. With that in mind, designers can use the power of Excel formulas to 
build the simulation. These spreadsheets can sort or make calculations on the time 
interval data that is received from the system dynamics model, which allows the 
ability to show trends. This fimctionality allows a large amount of the calculations 
and number-crunching to occur in the spreadsheet and not in the code, placing more 
control with the activity designers. 
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The different cells in the spreadsheet model can be configured as parameter inputs or 
parameter outputs for a system dynamics model. This is what the system dynamics 
engine uses to write and read data from the system dynamics model, pass values to 
the ICA and update the student's work in the on-line activities. By making the 
5 spreadsheet object the central repository for the system dynamics data, we ensure 
that the system dynamics model, simulation model, activity and ICA are all in 
synch. 

The system dynamics model generates simulation results over time, based on 
relationships between the parameters passed into it and other variables in the system. 
A system dynamics object is used to integrate with Visual Basic and the spreadsheet 
object. The object includes logic that controls the time periods as well as read and 
write parameters to the system dynamics model. With Visual Basic, we can pass 
these parameters to and from the model via the values in the simulation object. 

The system dynamics object also controls the execution of the system dynamics 
model. What this means is that after all of the parameter inputs are passed to the 
system dynamics model, the engine can run the model to get the parameter outputs. 
The system dynamics object allows for the system dynamics models to execute one 
step at a time, all at once, or any fixed number of time periods. 

When the system dynamics model runs, each step of the parameter input and 
parameter output data is written to a 'backup* sheet for two reasons. First, the range 
of data that is received over time (the model playing multiple times) can be used to 
25 create trend graphs or used to calculate statistical values. Second, the system 

dynamics model can be restarted and this audit trail of data can be transmitted into 
the model up to a specific point in time. What this means is that the engine can be 
used to play a simulation back in tune. 

30 When any event occurs within the system dynamics engine, a log is created that tells 
the designers what values are passed to the simulation model, system dynamics 
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model and ICA as well as the current time and the event that occurred. The log is 
called "SysDyn.log*' and is created in the same location as the application using the 
engine. As with the spreadsheet object, the system dynamics object allows a large 
amount of the calculations to occur in the system dynamics model and not in the 
5 activity code, again placing more control with the activity designers. 

Model objects are used to configure the system dynamics models with regard to the 
time periods played. Models are what the parameter inputs and parameter outputs 
(discussed later) relate to, so these must be created first. Every model has the 



1 0 following appUcation progranmiing interface: 





Field Name 


Data Type 


Description 




ModellD 


Long 


Primary Key for the table 




TaskID 


Long 


TaskID of the task associated with the model 


ri 


ModelName 


String*50 


Name of the model (informational purposes) 


WW? 


ModelDesc 


String*50 


Description of the model (informational 
purposes) 




SysDynModel 


String*50 


Filename of the actual system dynamics model 


%^ 


Start 


Long 


Start time to play modal 




Stop 


Long 


Stop time to play model 




Step 


Long 


Interval at which to play one model step and 
record data 



This information is stored in the model table of the simulation database 
(ICASim.mdb), All of the values that will need to be manually entered by the 
student that are passed into the system dynamics model are configured as parameter 
15 inputs (PInputs) objects. 

Every PInput has an interface as detailed below. 



Field Name 


Data Type 


Description 


PinputID 


long 


Primary Key for the table 
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TaskID 


long 


TaskE) of the task associated with the 
parameter input 


ModellD 


long 


ID of the model associated with the 
parameter input 


InputName 


string*50 


Name of the parameter input (informational 
purposes) 


InputDesc 


string*255 


Description (informational purposes) 


ReferenceName 


string*50 


Name of the spreadsheet cell associated 
with the parameter input^ 


SimReferenceName 


string* 50 


Name of the associated parameter in the 
system dynamics model 


TutorAware 


boolean 


Whether the ICA should be notified of any 
changes to the parameter input 


SourceltemlD 


long 


SourceltemlD of the parameter input 


TargetID 


long 


TargetID of the parameter input 


Row 


long 


Spreadsheet row number of the parameter 
input 

(used for speed optimization) 


Column 


long 


Spreadsheet column number of the 

parameter input 

(used for speed optimization) 


SheetName 


string*50 


Sheet name were the parameter input is 
located 

(used for speed optimization) 



All of this information is stored for every parameter input in the PInput table of the 
simulation database (ICASim.mdb). 
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Plnputs consist of one spreadsheet cell that can be populated by a designer at design 
time or by the GBS application at run time via the system dynamics engine object's 
methods. The purpose of the cell is to provide an entry point to the simulation and 
system dynamics models. An example of an entry point would be the interest rate 
parameter in the interest calculation example. The ICA is notified of any changes to 
the cell when an appropriate activity transpires. When the ICA is notified of a 
change two messages are sent to the ICA. The first is an ICANotifyDestroy 
message with the parameter input information i.e., SourceltemID, TargetID and null 
as an attribute. This message is sent to inform the ICA to remove information fi-om 
its memory. The second message is an ICANotifyCreate message with the 
parameter input information i.e., SourceltemID, TargetID, Attribute (cell numeric 
value). This message advises the ICA to add this information to its memory. 

To demonstrate the use of a parameter input, the interest rate calculation example is 
again used as a backdrop to illuminate various features. Figures 49 is a PInput Cell 
for a simulation model in accordance with a preferred embodiment. Figure 50 is a 
PInput backup cell in a simulation model in accordance with a preferred 
embodiment. Interest Rate is the parameter input for this model which will then be 
used to calculate balance and interest accumulations. A defined name will also have 
to be created for the backup of the PInput as each time interval is played. A 
requirement for this cell is that it has the same name as the original PInput, but also 
have the "_Bir' extension. The example here would be "Interest_Rate_BU." This 
cell will also have to be created in a column where no other data exists , since all of 
the backups are vmtten below this cell. In the ICA, define a task that will be 
assigned to the simulation. For example, a TaskID of 123 is generated by the ICA. 
For this example, we will assume that we want to give feedback on the interest rate 
selected by the student. In the ICA, define a Target for the parameter input. 



1 PowerSim allows designers to create parameters as arrays. If this is the case, then each 
array item MUST have one parameter input. What this means is that dynamics arrays can 
not be used by the System Djrnamics engine. 
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A PInput table record in accordance with a preferred embodiment is presented 
below. 



PInputID: 


12345 


TaskID: 


123 


ModellD: 


1 


InputName: 


Interest Rate input 


InputDesc: 


Interest Rate input into interest 




calculation model 


ReferenceName: 


Interest_Rate 


SimReferenceName 


Param_Interest_Rate 


TutorAware: 


True 


SourceltemlD 


1201 


TargetED: 


4001 


Row: 


6 


Column: 


3 


SheetName: 


Sheetl 



Once the configuration is completed, the designer can also use the ICA Utilities to 
test the simulation. The Row, Column and SheetName values are automatically 
populated when the designer runs the parameters in the System Dynamics 
Workbench in the ICA Utilities. The reason for obtaining the cell coordinates is 
that the processing of cell data is significantly faster with cell positions than simply 
using the defined name. The system dynamics engine decodes the defined name 
(Reference Name) that the designer entered, and populates the table accordingly. 
This is an important step because there have been occasions when a designer would 
change the layout of a spreadsheet, i.e., move a defined name location, and then 
forget to perform this step. As such, bizarre data was being passed to the tutor; 
whatever data happened to reside in the old row and column. Cells in the 
spreadsheet that are the output firom a system dynamics models can be represented 
by parameter output objects (POutputs). Every POutput has an interface as detailed 
below. 
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Field Name 


Data Type 


Description 


PoutputID 


Long 


Primary Key for the table 


TaskID 


Long 


TaskID of the task associated with the parameter 
output 


ModellD 


Long 


ID of the model associated with the parameter output 


OutputName 


String*50 


Name of the parameter output (informational 
purposes) 


OutputDesc 


String*255 


Description (informational purposes) 


ReferenceName 


String*50 


Name of the spreadsheet cell associated with the 
parameter output 


SimReferenceName 


String*50 


Name of the associated parameter in the system 
dynamics model 


TutorAware 


Boolean 


Whether the ICA should be notified of any changes 
to the parameter output 


SourceltemID 


Long 


SourceltemID of the parameter output 


TargetID 


Long 


TargetID of the parameter output 


Row 


Long 


Spreadsheet row number of the parameter output 
(used for speed optimization) 


Column 


Long 


Spreadsheet column number of the parameter output 
(used for speed optimization) 


SheetName 


String*50 


Sheet name were the parameter output is located 
(used for speed optimization) 



All of this information is stored for every output in the Output table of the 
simulation database (ICASim.mdb). Each POutput object conprises a spreadsheet 
cell that is an output from the system dynamics model. This is the information that 
shows the student the results of their choices and is the main reason for using system 
dynamics. The POutput can be made TutorAware, which will notify the ICA of any 
changes to the cell when all the POutputs are processed otherwise the ICA will be 
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unaware of any changes. When the ICA is notified of a change, two messages are in 
fact sent to the ICA: 

1 . An ICANotifyDestroy message with the parameter output information i.e., 
SourceltemID, TargetID and null as Attribute. This message is to advise the ICA to 

5 remove this information firom its memory. 

2. An ICANotifyCreate message with the parameter output information i.e., 
SourceltemID, TargetID, Attribute (cell numeric value) . This message is to advise 
the ICA to add this information to its memory. 

As opposed to PInputs which notify the ICA on every change, POutputs are 
1 0 processed in batch just before asking the ICA for feedback . 

POutputs use is illuminated below by an example that builds on the previous interest 
kO calculation example. Here, we want to notify the ICA of the balance as it results 

i'b from changes in the interest rate. Figure 51 is a display illustrating a POutput cell in 

15 accordance with a preferred embodiment. The steps required to configixre the 
ifl POutput are presented below. 

■■^ 

L Defme a name for cell G4 in Excel Here we have defined "Balance." 
m 2. Define the name of the backup cell as "BalanceJBU" in a blank column. 

^ 20 3. Let's use the same TaskID as before since the Balance parameter is part of the 

same simulation as the Interest Rate parameter. Ex: TaskID is 123. 

4. In the ICA, define a Target for the parameter output. Ex: a TargetID of 4005 is 
generated by the ICA. 

5. In the ICA, define a Sourceltem for the parameter output. Ex: a SourceltemID of 
25 1 2 1 5 is generated by the ICA. 

6. Associate the parameter output to a system dynamics model (refer to Model 
object discussion). 

7. Add the information in the POutput table of the simulation engine database. This 
configuration can also be done within the ICA Utilities. 

30 

The record in the POutput table would look something hke this: 
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OutputID: 


12347 


TaskID: 


123 


ModellD: 


1234 


OutputName: 


Balance Value 


OutputDesc: 


Value of Balance after model has 
been run 


ReferenceName: 


Balance 


SimReferenceName 


Param_Balance 


TutorAware: 


True 


SourceltemlD 


1215 


TargetID: 


4005 


Row: 


4 


Column: 


7 


SheetName: 


Sheetl 



o 

IB 
11 

m 
m 
u 
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The following information provides details describing the interaction components in 
accordance with a preferred embodiment. 



Title 


Description 


Procedural tasks (w/drag 
drop) 


Tasks which require the construction of some kind of report 
with evidence dragged and dropped to justify conclusions 


Procedural tasks (w/o drag 
drop) 


New task designs that are procedural in nature, have very 
little branching, and always have a correct answer. 


Ding Dong task 


Tasks that interrupt the student while working on something 
else. This template includes interviewing to determine the 
problem, and a simple checkbox form to decide how to 
respond to the situation. 


Analyze and Decide (ANDIE) 
task 


Most commonly used for static root cause analysis, or 
identification tasks. Developed on SBPC as a result of 3 
projects of experience redesigning for the same skill 


Evaluate Options (ADVISE) 


Used for tasks that require learner to evaluate how different 
options meet stated goals or requirements. Developed at 
SBPC after 4 projects experience redesigning for the same 
skill. Does not allow drag drop as evidence. 


Run a company task 


Time based simulation where student "chooses own 
adventure". Each period the student selects from a pre- 
determined hst of actions to take. Developed on SBPC as a 
simplified version of the BDM manage task. 


Use a model task 


When user needs to interact with a quantitative model to 
perform what if analysis. May be used for dynamic root 
cause analysis - running tests on a part to analyze stress 
points. 


IC A Dynamic Meeting Task 


Developed on BDM to mimic interaction styles from Coach 
and ILS EPA. Supports dynamic-rule based branching - will 
scale to support interactions like EnCORE defense meetings 
and YES. 
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Manage Task 


Time based simulation where student manages resources. 
Human Resources Management, managing a budget, manage 
an FX portfolio. 


QVID Static Meeting Task 


Developed on Sim2 to support agenda-driven meetings 
where user is presented with up to 5 levels of follow-up 
questions to pursue a line of questioning. As they ask each 
question, it's follow-ups appear. 


Flow Chart Task 


Will support most VISIO diagrams. Developed on Sim2 to 
support simple flow chart decision models. 


QVID Gather Data 
Component 


Static flat list of questions to ask when interviewing 
someone. Not used when interviewing skills are being 
taught (use QVID Static meeting task). Supports 
hierarchical questions and timed transcripts. 


JoumaUze Task 


Created to support simple joumal entry tasks with up to 2 
accounts per debit or credit. 


New Complex Task 


A new task that requires a simulation component 
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Systems Dynamic Engine 

The system dynamics engine is the interface between the simulation model, the 
system dynamics model, the simulation database and the Intelligent Coaching 
Agent. The system dynamics engine is of interest to the designer so that she can 
understand the mechanics of it. Once the designer has constructed the simulation 
model (Excel Spreadsheet), built the system dynamics model (PowerSim) and 
configured all of the parameter inputs and parameter outputs, a test can be performed 
using the workbench included in the IC A UtiUties (refer to ICA Utilities 
docimientation). The developers, in turn, need to implement the calls to the system 
dynamics engine in the GBS apphcation that is being built. The following list 
identifies the files that need to be included in the Visual Basic project to use the 
system dynamics engine. 



WSysDynEng.cls 


System dynamics Engine class 


wSysDynEng.bas 


System dynamics Engine module (this module was 
introduced only for speed purposes because all the code 
should theoretically be encapsulated in the class) 


wConst.bas 


Intelligent Coaching Agent constant declaration 


wDeclare.bas 


Intelligent Coaching Agent DLL interface 


wlca.cls 


Intelligent Coaching Agent class 


wica.bas 


Intelligent Coaching Agent module (this module was 
introduced only for speed purposes because all of the code 
should theoretically be encapsulated in the class) 



To utilize the system dynamics engine fully, the developer must place code in 
different strategic areas or stages of the application. 

1) Initial stage - the loading of the form containing the simulation front-end. This 
is when the simulation model and system dynamic engine are initialized. 
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2) Modification stage - Takes place when the user makes changes to the front-end 
that impacts the simulation model PInputs), This is when the ICA is notified of 
what's happening. 

3) Run stage - The system dynamics model is run and parameter outputs are 
received. 

4) Feedback stage - The user requests feedback on the work that they have 
performed. This is when the simulation notifies the ICA of all output changes. 

5) Final stage - The simulation front-end unloads. This is when the simulation 
model is saved. 

These stages will be explained by including the Visual Basic code involved as well 
as a short description of that code. 

Initial Stage Code In Accordance With A Preferred Embodiment 

1. Creating the ICA & the simulation engine objects 

Code: 

Set moSysDynEngine = New classSysDynEngine 
Set moICA = New classICA 

Description: The first step in using the system dynamics engine is to create an 
instance of the classSysDynEngine class and also an instance of the classICA class. 
Note that the engine and ICA should be module level object "mo" variables. 

2. Loading the simulation 

Code: 

IRet = moSysDynEngine. OpenSimulation(FILE_SIM, Me.bookSim, True) 
IRet = moSysDynEngine.LoadSysDyn(mlICATaskID, DB_SIMULATION, 1) 
IRet = moSysDynEngine.LoadModel(MODEL_NAME,mbTaskStarted) 

Description: After the object creation, the OpenSimulation, LoadSimulation and 
LoadModel methods of the system dynamics engine object must be called. The 
OpenSimulation method reads the specified Excel 5.0 spreadsheet file (FILE__SIM) 



